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Abstract 

The  problem  in  this  study  is  to  determine  whether  Air  Force  contract  protest 
frequency  rises  or  falls  with  reforms  in  the  acquisition  process,  number  of  AF  Contract 
Actions  and  Dollars,  AF  GAO  Sustain  Rate  and/or  General  Economic  Conditions.  The 
study  outlines  the  possible  relationships  between  the  independent  variables  (previously 
listed)  and  the  dependent  variable  (protests).  The  number  of  AF  GAO-Protests  Processed 
was  used  to  represent  the  number  of  protests;  therefore,  care  must  to  taken  in  generalizing 
results  to  other  protest  forums  or  services.  Multiple  regression  techniques  were 
employed  to  test  the  statistical  significance  of  the  models’  response  and  predictor 
variable(s).  The  results  indicated  that  AF  Contract  Actions,  Dollars,  and  General 
Economic  Conditions  were  statistically  significant  as  a  predictor  of  AF  GAO-Protests 
Processed;  however,  the  GAO  Sustain  rate  was  not  useful.  Incorporating  the  qualitative 
variable,  the  researcher  determined  that  the  FASA/FAR  15  (efficiency  reform  years)  were 
not  statistically  significant  to  the  model.  In  conclusion,  the  study  points  to  the  complex 
dynamics  associated  with  measuring  trends  with  the  number  of  protests  associated  with  a 
particular  variable.  Secondly,  the  acquisition  reforms  of  the  1990s  have  not  been  in 
existence  long  enough  to  be  tested  with  a  high  degree  of  reliability.  Implications  for  the 
AF  decision-maker  and  professional  are  discussed  and  recommendations  for  future  study 
are  presented.  Air  Force  decision-makers  will  be  able  to  use  the  results  of  this  study  as 
part  of  an  overall  assessment  of  acquisition  reform  and  the  bid  protest  process. 
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THE  IMPACT  OF  ACQUISITION  REFORM  AND  POLITICAL-FISCAL 
VARIABLES  ON  AIR  FORCE  GAO-PROTESTS  PROCESSED 


I.  Introduction 


Problem 

Federal  contractors  expend  a  great  deal  of  resources  in  the  preparation  of 
bids/offers  in  response  to  an  Air  Force  solicitation.  In  this  competitive  and  highly 
volatile  environment,  AF  agencies  strive  to  maintain  the  integrity  of  the  acquisition 
process  through  a  fair  evaluation  of  all  responsive  bids/offers.  If  a  perspective 
offeror/bidder  perceives  the  slightest  inequity  in  the  process,  they  may  exercise  the  right 
to  legally  challenge  the  award  of  an  Air  Force  contract.  The  likelihood  of  a  protest  may 
be  influenced  by  acquisition  reform  initiatives,  size  of  the  AF  procurement  budget, 
number  of  contract  actions  and/or  general  economic  health.  While  these  external  factors 
may  be  out  of  the  acquisition  professional’s  control,  an  understanding  of  complex  and’ 
dynamic  relationships  will  provide  AF  decision-makers  a  tool  to  identify  trends  in 
contract  protest  frequency. 

Importance  of  Research.  As  the  number  of  protests  received  by  GAO  exceeds 
2,500  annually  (Metzger  and  Golden,  1997:  viii),  acquisition  professionals  should  be 
concerned  about  the  bid  protest’s  significance.  Such  actions  involve  attorney  fees, 
administrative  costs,  witness  fees,  and  filing  fees  incurred  by  the  Government  and 
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contractor  (Duncan,  1997:  9).  In  addition,  contract  performance  may  be  suspended 
during  protests,  preventing  timely  delivery  and  satisfaction  of  Government  requirements 
(Duncan,  1997:  9).  When  these  delays  involve  major  programs,  the  resultant  costs  could 
conceivably  amount  to  millions  of  dollars  (Duncan,  1997:  9). 

Recent  DOD  examples  highlight  the  explicit  and  implicit  costs  associated  with  a 
protest.  A  $673M  Navy  TAC-4  Computer  Work  Station  contract  award  was  held  up  by  a 
single  disappointed  bidder  (Electronic  News,  1996:  2).  A  contractor’s  protest  of  a 
$23. 5M  satellite  component  contract  forced  the  Air  Force  to  halt  phase  II  of  an 
acquisition  program  (Electronic  News,  1995:  1).  The  delays  and  costs  represent  a  great 
burden  on  the  public  and  private  sector.  In  an  effort  to  improve  Government-contractor 
business  relations,  acquisition  reforms  have  targeted  the  bid  protest  mechanism,  post¬ 
award  debriefing  and  source  selection  procedures.  Various  reforms  between  1984-1996 
have  emphasized  improved  communication,  increased  competition,  and  contracting 
officer  autonomy. 

Background.  When  Congress  made  the  General  Accounting  Office  (GAO) 
procurement  protest  system  part  of  the  Competition  in  Contracting  Act  (CICA)  of  1984, 
they  believed  that  “a  strong  enforcement  mechanism  is  necessary  to  insure  that  the 
mandate  for  competition  is  enforced  and  that  vendors  wrongfully  excluded  from 
competing  for  Government  contracts  receive  equitable  relief’  (Cobum,  1985: 47).  The 
Federal  Acquisition  Streamlining  Act  of  1994  (FAS A)  instituted  various  reforms  to  make 
the  federal  procurement  system  more  efficient,  more  accessible,  and  more  competitive 
(Gamboa  and  others,  1997:  3).  By  addressing  the  fair  competition  issue,  the  law  should 
have  decreased  the  number  of  protests. 
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The  number  of  new  protests  fell  by  18%  in  the  first  half  of  FY  1998;  however,  the 
GAO’s  sustain  rate  rose  from  12%  to  18%  (Gordon,  1998:  5).  In  FY  97,  the  top  reasons 
for  protests  in  the  Air  Force  were  improper  evaluation,  challenge  of  awards, 
defective/restrictive  specifications,  and  exclusion  from  the  competitive  range  (SAF/AQC, 
1998:1).  Protests  may  be  declining  because  source  selection  officials  are  doing  a  better 
job  with  their  procurements;  however,  the  trends  could  be  related  to  other  external  and 
internal  factors  (Gamboa,  1998:  4).  There  are  many  external  factors  that  play  a  role  in  a 
contractor’s  decision  to  protest,  such  as  financial  health  of  the  firm,  size  of  the 
Department  of  Defense  budget,  and  overall  economic  conditions  (Thomas,  1997:  17-18). 

In  order  to  make  an  accurate  assessment  of  acquisition  reform’s  effect  on  bid 
protests,  the  Air  Force  needs  to  take  an  objective  look  at  protest  trends  and  associated 
conditions.  A  historical  analysis  that  culminates  with  statistical  results  may  shed  light  on 
the  utility  of  continued  reforms  in  the  bid  protest  arena.  While  the  results  of  the  research 
will  not  establish  causal  relationships,  the  statistical  correlation  will  provide  Air  Force 
decision-makers  a  tool  to  identify  trends  in  contract  protest  frequency. 

Management  Questions 

Management  questions  surface  regarding  acquisition  reform  and  external  factors 
related  to  the  number  of  protests  in  the  AF.  Have  acquisition  reform  initiatives  affected 
the  number  of  protests  filed?  Do  other  external  factors  influence  the  amount  of  protests 
filed?  How  much  have  acquisition  reform  initiatives  affected  the  number  of  protests  filed 
relative  to  external  factors? 
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Research  Objectives 


The  primary  objective  of  this  research  is  to  determine  whether  acquisition  reform 
measures  have  made  an  impact  on  the  number  of  protests  filed.  Since  a  number  of 
professionals  point  to  other  external  conditions  that  influence  the  protest  decision,  other 
external  factors  will  be  introduced  as  variables.  Consequently,  another  research  objective 
will  be  to  determine  if  the  number  of  contract  actions,  amount  of  contract  dollars 
awarded,  and  GAO  sustain  rate  affected  the  number  of  protests.  Finally,  the  research  will 
analyze  contractor  protests  with  relation  to  general  economic  health  (represented  by  the 
unemployment  rate). 

The  objectives  are  listed  below: 

•  Identify  trends  in  the  number  of  protests  filed  from  1984-1 998 . 

•  Identify  the  protest  trends  associated  with  CICA,  FAS  A  and  FAR  1 5  rewrite. 

•  Identify  the  protest  trends  associated  with  the  AF  Contract  actions  and  Dollars 
awarded. 

•  Identify  the  protest  trends  associated  with  general  economic  conditions 
(unemployment  rate). 

•  Determine  if  a  significant  statistical  relationship  exists  between  acquisition  reform 
and  the  number  of  protests  filed  in  the  AF. 

•  Determine  if  a  significant  statistical  relationship  exists  between  the  contract  actions, 
dollars  awarded  and  general  economic  conditions  with  the  number  AF  GAO-Protests 
Processed. 

•  Test  and  evaluate  the  following  factors  using  multiple  regression  analysis: 

a)  Dependent  Variable:  AF  GAO-Protests  Processed  for  a  particular  year 
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b)  Independent  Variable  (Quantitative):  AF  Contract  Actions 

c)  Independent  Variable  (Quantitative):  AF  Contract  Dollars 

d)  Independent  Variable  (Quantitative):  AF-GAO  Sustain  Rate 

e)  Independent  Variable  (Quantitative):  General  Economic  Conditions 

f)  Independent  Variable  (Qualitative):  Acquisition  reform  (CICA,  FASA,  FAR 
15) 

Further  description  of  the  coding  mechanism  and  multiple  regression  techniques  will 
appear  in  the  Methodology  Chapter. 

Scone  and  Limitations 

This  research  focuses  on  the  protests  filed  at  the  General  Accounting  Office 
(GAO)  during  the  post  and  pre  award  stage  of  Air  Force  contracts  (hereby  referred  to  as 
Air  Force-GAO  Protests  Processed).  It  tracks  and  analyzes  the  number  of  protests  filed 
from  1984  to  1998  as  related  to  CICA,  FASA,  and  the  FAR  15  rewrite.  In  addition,  the 
number  of  contract  actions,  contract  dollars,  GAO  sustain  rate,  and  the  unemployment 
rate  are  compared  to  the  respective  year.  From  the  data,  the  research  attempts  to 
determine  the  effects  of  the  listed  variables  and  assesses  statistical  correlation  and 
relationships. 

The  study  involves  quantitative  analysis.  The  purpose  is  to  provide  a  broad 
statistical  analysis  of  the  number  of  Air  Force-GAO  Protests  Processed  as  correlated  to 
the  variables.  To  ensure  the  validity  and  accuracy  of  the  data,  SAF/AQC  has  provided 
the  guidance  and  numbers  used  in  the  study.  A  primary  purpose  is  to  determine  if 
acquisition  reform  initiatives  make  a  statistically  significant  impact  on  the  number  of 
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protests  filed.  Since  multiple  regression  analysis  is  used,  extreme  caution  is  exercised  in 
interpreting  the  results  of  the  analysis.  Correlation  is  not  causation;  therefore,  the  results 
are  limited  to  the  appropriate  level  of  meaning  for  this  type  of  statistical  analysis. 

Summary 

This  chapter  provided  a  general  overview  of  the  bid  protest  process  and  changes 
related  to  acquisition  reform.  In  addition,  it  introduced  other  external  factors  that  may 
influence  the  likelihood  of  a  contractor  filing  a  protest.  Most  importantly,  the  chapter 
stated  the  research  problem,  importance,  management  questions,  research  objectives, 
scope  and  limitations  of  the  study.  The  next  chapter  reviews  the  literature  that  explores 
the  evolution  of  the  bid  protest  process  as  related  to  acquisition  reform,  previous  research 
efforts  in  the  bid  protest  area,  and  justification  for  the  use  of  stated  external  factors. 
Chapter  III,  Methodology,  explains  the  data  collection  and  quantitative  analysis  methods 
used  in  this  research.  Chapter  IV  consists  of  the  data  analysis.  Chapter  V  synthesizes  the 
results  to  draw  meaningful  conclusions  for  the  AF  decision-maker  and  suggest  future 
areas  of  study. 
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II.  Literature  Review 


Introduction 

This  chapter  reports  and  investigates  literature  in  the  area  of  the  bid  protest 
process,  reform  of  the  acquisition  laws  as  related  to  bid  protests,  costs  associated  with 
protests,  and  other  factors  that  may  influence  the  number  of  protests.  A  definition  of  key 
terms  related  to  the  bid  protest  process  is  followed  by  an  in-depth  analysis  of  the 
evolution  of  laws  and  regulations  that  govern  the  process.  The  chapter  reports  the 
strengths  and  weaknesses  of  previous  research  in  the  bid  protest  arena.  Another 
paradigm  related  to  acquisition  reform’s  effectiveness  is  introduced  to  uncover  potential 
variables  (contract  actions,  contract  dollars,  GAO  sustain  rate,  unemployment  rate)  that 
may  influence  the  decision  to  protest.  The  chapter  concludes  with  a  focus  on  the  current 
study. 


Protests  Defined 

A  substantial  part  of  the  procurement  process,  contract  award  controversies 
between  government  agencies  and  offerors,  prospective  offerors,  and  other  parties,  often 
lead  to  formal  protests  (Cibinic  and  Nash,  1998:  1481).  According  to  the  Federal 
Acquisition  Regulation  (FAR),  a  protest  “means  a  written  objection  by  an  interested  party 
to  any  one  of  the  following: 

(a)  A  solicitation  or  other  request  by  an  agency  for  offers  for  a  contract  for  the 
procurement  of  property  or  services. 

(b)  The  cancellation  of  the  solicitation  or  other  request. 

(c)  An  award  or  proposed  award  of  the  contract. 

(d)  A  termination  or  cancellation  of  an  award  of  the  contract,  if  the  written 
objection  contains  an  allegation  that  the  termination  or  cancellation  is  based  in 
whole  or  in  part  on  improprieties  concerning  the  award  of  a  contract.  (FAR 
33.101) 
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Only  protests  submitted  by  an  “interested  party”  that  relate  to  a  particular 
solicitation  or  the  proposed  award  or  award  of  a  particular  contract  will  be  heard  by  the 
Comptroller  General  (Cibinic  and  Nash,  1998:  1496).  Regarding  a  protest,  the  FAR 
defines  an  interested  party  as  the  actual  or  prospective  bidder/offeror  whose  direct 
economic  interest  would  be  affected  by  the  agency’s  contract  award  or  failure  to  award 
(FAR  33. 1 01).  To  allege  a  direct  economic  interest,  the  actual  or  prospective 
bidder/offeror  must  be  next  in  line  for  award  or  be  competitive  for  award  if  their  protest 
is  sustained  (Cibinic  and  Nash,  1998:  1498).  A  protest  can  be  “sustained”  or  “denied”  by 
the  GAO  or  agency,  or  withdrawn  by  the  contractor. 

The  bid  protest  system  exists  in  the  unique  area  of  public  sector  procurement  to 
protect  the  integrity  of  the  procurement  process  and  safeguard  protestor  rights  (Cantor, 
1997:  155).  Since  the  government  is  often  the  largest  and  only  buyer  in  certain  markets 
and  public/private  sector  differences,  the  bid  protest  system  imposes  a  unique  set  of  rules 
on  the  contractors  (Cantor,  1997:  155).  The  protestor  faces  a  number  of  forums  to  obtain 
relief:  the  General  Accounting  Office  (GAO),  a  federal  district  court,  the  Court  of  Federal 
Claims,  or  the  procuring  agency.  The  GAO,  the  oldest  of  the  forums,  has  developed  into 
the  most  popular  choice  for  the  protestors  (aside  from  agency  procedures)  and  has 
resolved  the  greatest  number  of  protests  in  comparison  to  the  other  forums  (Cantor,  1997: 
158). 

While  the  protestor  may  file  with  the  agency  or  GAO,  the  interested  party  is 
encouraged  to  seek  resolution  with  agency  before  filing  to  the  GAO  (FAR  33.102). 
Clearly,  protests  to  the  agency  are  encouraged  to  reduce  the  formality,  costs,  complexity, 
and  time  in  a  fair  resolution  of  the  dispute.  On  the  other  hand,  the  protests  to  the  GAO 
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may  involve  a  statutory  stay  of  award,  increased  costs,  a  higher  level  of  formality,  and 
rigid  time  constraints.  For  these  reasons,  the  research  will  focus  on  the  Air  Force-GAO 
Protests  Processed. 

Monetary  and  Nonmonetary  Costs 

At  the  beginning  of  the  protest  process,  the  “statutory  stay”  provision  delays 
award  or  suspends  contract  performance  pending  a  decision  on  a  protest.  If  filed  before 
contract  award,  the  protest  precludes  the  award  of  the  contract  until  the  matter  is 
resolved.  If  filed  after  contract  award,  the  protest  will  cause  the  federal  agency  to 
“immediately  direct  the  contractor  to  cease  performance  under  the  contract”  (Cibinic  and 
Nash,  1998: 1506).  Consequently,  Air  Force  agencies  will  be  forced  to  devote  resources 
to  resolve  the  protest  and  experience  a  delay  and  risk  cancellation  of  the  procurement. 
This  study  will  cover  both  the  preaward  and  postaward  protests  filed  at  the  GAO  on  AF 
acquisitions. 

In  addition  to  the  wasted  resources  and  threat  to  the  procurement,  the  protest  can 
result  in  other  nonmonetary  remedies.  The  Comptroller  General  may  recommend  that  the 
Federal  agency:  (1)  refrain  from  exercising  options  on  a  contract,  (2)  recompete  the 
contract,  (3)  re-issue  the  solicitation,  (4)  terminate  the  contract,  (5)  award  a  contract 
consistent  with  statutory  requirements,  and/or  (6)  implement  a  combination  of  the  listed 
measures  (Cibinic  and  Nash,  1998:  1529).  Each  of  these  nonmonetary  remedies  would 
cause  a  great  deal  of  opportunity  cost  to  the  AF  agencies  in  reaccomplishing  and/or 
canceling  preaward  activities. 
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With  regard  to  monetary  remedies,  the  GAO  may  determine  that  the  agency  pay 
the  following  costs  as  delineated  by  the  FAR: 

[Government  may  pay]  the  cost,  exclusive  or  profit,  of  filing  and  pursuing  the 
protest,  including  reasonable  attorney,  consultant,  and  expert  witness  fees,  and  bid 
and  proposal  preparation  costs.  The  agency  shall  use  funds  available  for  the 
procurement  to  pay  the  costs  awarded.  (FAR  33.104) 

Since  the  actions  of  a  disgruntled  bidder/offeror  may  result  in  the  payment  of  money, 
agencies  must  identify  the  factors  and  circumstances  conducive  to  the  filing  of  a  GAO 
protest. 

Competition  in  Contracting  Act  of  1984  ('CICAt 

Until  1984,  The  General  Accounting  Office  (GAO)  was  the  only  forum  for 
disappointed  bidders/offerors  to  lodge  contract  protests  against  government  agencies 
until.  The  Comptroller  General  issued  decisions  under  the  statutory  authority  of  31 
U.S.C.  which  allowed  for  the  settlement  and  adjustment  of  “accountable  officers”  and 
decisions  concerning  the  “legality  of  payments”  (Cibinic  and  Nash,  1998: 1491).  As  the 
procurement  agencies  generally  agreed  with  the  decisions,  the  GAO  Protest  Processing 
mechanism  became  established  in  the  system. 

The  informal  system  accepted  by  contractors  and  government  agencies 
encountered  a  watershed  in  1984.  In  that  year,  the  Comptroller  General  was  given  direct 
statutory  authority  to  hear  protests  and  order  a  stay  of  performance  (Cibinic  and  Nash, 
1998:  1492).  Congress  enacted  the  General  Accounting  Office  (GAO)  as  part  of  the 
Competition  in  Contracting  Act  (CICA)  in  1984.  These  major  changes  to  the 
administrative  bid  protest  system  took  effect  on  April  1,  1985  (Cobum,  1985:  47).  The 
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new  law  addressed  perceived  deficiencies  in  the  GAO  bid  protest  remedy:  (1)  denial  of 

relief  for  meritorious  protest,  (2)  lack  of  stay  of  contract  performance,  and  (3)  lengthy 

GAO  review  process  (Cobum,  1985:  48).  In  addition,  CICA  gave  contractors  substantial 

new  grounds  to  protest  contract  award  (competition  requirements,  advertising 

requirements,  defective  specifications)  (Cobum,  1985:  54). 

In  addition  to  the  full  and  open  competition  requirement  imposed  upon  the 

agencies,  the  contractors  achieved  a  great  degree  of  power  in  their  quest  for  federal 

dollars.  As  an  interested  party,  a  perspective  bidder  could  file  a  protest  and  delay  the 

award  of  a  federal  contract  until  the  issue  is  resolved.  A  prominent  contract  attorney 

displayed  the  general  attitude  prevalent  after  the  enactment  of  CICA: 

Protesting  an  improper  bid  award  demonstrates  to  the  awarding 
agencies  that  they  must  play  by  their  own  rules.  One  thing  is  certain  in 
dealing  with  government  agencies:  If  they  have  learned  you’ll  accept 
being  treated  unfairly  without  a  whimper,  they’ll  do  it  as  often  as  they 
wish. . . .each  time  you  fight,  no  matter  what  the  result,  the  next  award  is 
more  likely  to  be  given  to  your  company.  (Garofalo,  1 990:  95) 

At  the  cost  of  a  postage  stamp,  contractors  were  able  to  exercise  their  right  to 
protest  on  substantial  new  grounds  under  CICA.  The  failure  to  adhere  to  the  time 
advertisement  requirement  for  a  solicitation  or  meet  the  criteria  to  clearly  describe  agency 
needs  in  the  Commerce  Business  Daily  (CBD)  could  warrant  a  protest.  Secondly,  an 
agency  had  to  “specify  its  needs  and  solicit  bids  and  proposals  in  a  manner  designed  to 
achieve  full  and  open  competition  for  procurement”  (Cobum,  1985:  54).  Next,  an  agency 
failure  to  comply  with  the  statutory  limits  on  establishing  or  forcing  bidder  qualification 
requirements  would  be  grounds  for  a  protest  (Cobum,  1985:  54).  Finally,  the  failure  of  a 
procuring  agency  to  act  on  the  recommendation  of  the  Small  Business  Administration 
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(SB A)  representative  to  broaden  a  competitive  procurement  could  bring  a  protest 
(Cobum,  1985:  54).  Along  with  the  traditional  grounds  for  protest,  these  new  issues 
would  test  the  constitutional  limits  of  CICA  and  discover  if  the  full  and  open  atmosphere 
would  result  in  fewer  protests. 

Federal  Acquisition  Streamlining  Act  of  1994  fFASAl 

Many  acquisition  professionals  and  lawmakers  consider  the  Federal  Acquisition 
Streamlining  Act  of  1 994  (FAS A)  to  be  the  most  far  reaching  statutory  change  in  the  bid 
protest  process  since  CICA.  The  Federal  Acquisition  Streamlining  Act  (FASA),  enacted 
October  13, 1994,  represents  the  first  significant  statutory  change  in  the  bid  protest 
process  since  CICA  (Adelson,  1996:  5).  FASA  imposed  a  requirement  for  more 
substantial  post-award  debriefings  for  unsuccessful  offerors,  provided  for  the  use  of 
protective  orders,  and  changed  the  way  critical  time  limits  were  handled  (Adelson,  1996: 
6).  In  response  to  numerous  industry  complaints,  a  major  tenant  of  the  law  secured  the 
rights  of  disappointed  offerors  to  obtain  a  full  and  prompt  explanation  of  why  they  were 
not  awarded  a  contract  (FASA  Guide,  1995:  1-4).  Also,  the  new  law  showed 
Congressional  discontent  with  the  GAO  and  GSBCA  procedures  in  handling  protests 
(Adelson,  1996:  5). 

In  the  area  of  postaward  debriefings,  the  law  made  four  important  changes.  First, 
disappointed  offerors  must  be  notified  of  an  award  within  3  days  of  contract  award. 
Secondly,  awards  made  based  on  competitive  proposals  require  unsuccessful  offerors  to 
request  a  debriefing  within  3  days  of  notice  of  award.  Next,  this  debriefing  must  be 
provided  (to  the  maximum  extent  practicable)  within  5  days  of  agency  receipt  of  the 
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request.  Finally,  the  debriefing  must  contain  certain  minimum  requirements  (overall 
rankings  of  offerors,  rationale  for  award,  reasonable  responses  to  questions,  significant 
weaknesses  of  disappointed  offerors’  proposal  explained)  (FAS A  Guide,  1995: 1-5).  The 
overall  intent  of  the  new  debriefing  requirements  was  to  provide  the  disappointed  offerors 
with  timely  and  relevant  information  regarding  their  proposal.  Lawmakers  hoped  that 
this  would  eliminate  the  number  of  protests  filed  in  order  to  gain  information  concerning 
the  basis  of  contract  award  to  reduce  the  nonmonetary  and  monetary  costs  associated 
with  a  protest  (FASA  Guide,  1995: 1-5). 

FASA  ushered  in  changes  to  the  time  limits  and  procedural  processes  involved  in 
filing  a  protest.  While  the  formal  rules  and  strict  deadlines  have  always  been  present, 
FASA  offered  a  new  degree  of  flexibility  and  put  a  greater  degree  of  responsibility  on  the 
protestors  to  understand  the  time  limits  (Adelson,  1996:  8).  FASA  retained  the  rule  that 
the  protestor  had  10  days  from  the  date  the  basis  of  protest  is  known  to  file  a  protest; 
however,  the  GAO  implemented  the  rule  by  giving  protestors  14  calendar  days  (Adelson, 

1 996:  8).  About  a  suspension  of  a  procurement,  FASA  attempted  to  remove  the  incentive 
to  protest  to  stop  a  procurement.  It  allowed  the  GSBCA  to  dismiss  a  “frivolous”  or  “bad 
faith”  protest  and  linked  the  10-day  protest  limit  to  the  period  following  the  postaward 
debriefing  (Adelson,  1996:  9). 

With  the  dissolution  of  the  GSBCA  and  procedural  changes  in  the  GAO, 
contractors  were  faced  with  new  rules  and  regulations  that  balanced  fairness  and 
efficiency  in  the  bid  protest  process  (Fausti  and  Lee,  1997: 10).  President  Clinton  issued 
an  executive  order  that  encouraged  bidders  to  file  their  protests  with  the  agency  awarding 
the  contract  as  opposed  to  the  GAO.  If  the  issue  could  not  be  resolved  at  the  agency 
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level,  the  contractor  could  still  refile  the  protest  at  the  GAO.  In  an  effort  to  expedite  the 
process,  the  GAO  is  required  to  issue  a  decision  on  the  protest  no  later  than  1 00  days 
after  the  protest  is  filed  (Fausti  and  Lee,  1997:  1 1).  A  streamlined  and  efficient  protest 
system  that  discouraged  frivolous  protest  in  a  new  open  atmosphere  with  contractors 
should  have  affected  the  number  of  protests  filed. 

FAR  15  Rewrite  (19971 

The  FAR  Council  tasked  an  ad  hoc  interagency  committee  to  rewrite  FAR  1 5, 
Contracting  by  Negotiation  in  January  1996  and  the  final  rule  became  effective  on  10 
October  1997  (Federal  Register,  1997:  51224).  The  goals  of  the  rewrite  were  to 
incorporate  innovative  techniques  in  the  source  selection  process,  simplify  the  process, 
and  facilitate  acquisition  of  the  “best  value”  (Federal  Register,  1997:  51225).  To  attain 
these  goals,  the  FAR  15  rewrite  ushered  in  a  series  of  changes  that  expanded  the  need  for 
Contracting  Officer  judgement  in  the  equitable  treatment  of  contractors. 

Prior  to  the  FAR  15  rewrite,  the  standard  rule  for  competitive  range 
determinations  was  “when  there  is  a  doubt  as  to  whether  a  proposal  is  in  the  competitive 
range,  the  proposal  should  be  included”  (Federal  Register,  1997:  51227).  Consistent  with 
the  Clinger-Cohen  Act  of  1996,  the  revision  indicates  that  Contracting  Officers  should 
establish  a  competitive  range  of  the  most  “highly  rated  proposals”  (Federal  Register, 

1997:  51229).  The  objective  of  the  changes  were  twofold:  1)  contractors  with  a  good 
chance  of  winning  the  contract  would  compete  more  aggressively,  and  2)  those 
eliminated  from  the  range  would  avoid  wasting  resources  in  a  futile  effort  (Federal 
Register,  1997:  51229).  Contracting  Officers  gained  increased  flexibility  on  the 
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competitive  range  determination;  however,  they  needed  to  follow  reasonable  and  fair 
methods  to  make  such  a  determination.  Arbitrary  exclusions  from  the  competitive  range 
and  inconsistent  application  of  competitive  range  criteria  would  lead  to  contractor 
protests. 

The  FAR  15  rewrite  introduced  “efficient  competition”  as  the  guidance  to  apply 
the  criteria  for  advancing  into  the  competitive  range  (SAF/AQC,  1998).  While  the  final 
rule  did  not  provide  an  exact  definition  of  efficient  competition,  it  allowed  for  the  facts  of 
the  “instant  acquisition”  to  apply  (Federal  Register,  1997:  51227).  The  FAR  describes 
the  process  for  limiting  the  competitive  range  for  the  purpose  of  efficiency;  however,  the 
contracting  officer  is  free  to  exercise  authority  in  a  variety  of  circumstances  (Federal 
Register,  1997:  51229).  The  new  “when  in  doubt,  throw  them  out”  rule  could  provide  a 
level  of  discretion  to  contracting  officers  that  could  lead  to  abuse.  Offerors  might  be 
excluded  from  the  competitive  range  for  arbitrary  reasons  unrelated  to  the  actual 
procurement  (ad  hoc  blacklisting,  etc).  The  power  to  reduce  the  number  of  proposals  in 
the  competitive  range  to  the  “greatest  number  that  will  permit  an  efficient  competition 
among  the  offerors  rated  most  highly”  could  lead  to  an  increase  in  the  number  of  protests 
filed. 

The  FAR  15  rewrite  also  brought  significant  changes  in  the  types  of  exchanges 
and  communications  between  the  Government  and  contractor.  Negotiations  gave  the 
offeror  an  opportunity  to  revise  their  proposal  using  bargaining,  persuasion,  and  give  and 
take  after  establishing  the  competitive  range.  The  rule  contains  limits  on  exchanges  that 
preclude  favoring  one  offeror  over  another,  revealing  offeror’s  technical  solutions, 
revealing  prices  without  the  offeror’s  permission,  and  knowingly  furnishing  source 
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selection  information.  Expanded  exchanges  allowed  robust  discussions  having  offeror 
revisions  causing  the  relative  positions  of  offerors  to  change.  Increased  exchanges  with 
industry  increases  opportunity  for  unfair  competitive  advantage  or  disclosure  of  a 
potential  offeror’s  confidential  business  strategy. 

Acquisition  Reform  and  Protests:  The  Alternate  View 

Acquisition  reform  initiatives  have  attempted  to  increase  competition  and 
enhance  Government-contractor  communication  while  giving  the  contracting  officer 
increased  autonomy.  In  an  effort  to  foster  good  will,  many  of  the  reform  efforts  have 
attempted  to  address  the  reasons  influencing  contractor  protest.  Much  research  and  data 
relating  to  the  official  reasons  behind  a  protest  have  been  published.  On  the  other  hand, 
there  appears  to  be  a  lack  of  research  regarding  the  influence  of  acquisition  reform  (and 
external  factors)  on  the  number  of  protests  filed  in  the  Air  Force. 

The  acquisition  reform  initiatives  have  focused  on  modifying  the  behavior  of  the 
buyer  (government  agency)  in  an  attempt  to  improve  the  entire  process  (Dews  and 
Berkler,  1983: 2).  Lawmakers  believe  that  adopting  commercial  practices,  providing  a 
thorough  debriefing,  and  stating  requirements  in  a  way  to  provide  full  and  open 
competition  will  reduce  the  number  of  protests.  This  assertion  is  based  on  the 
assumption  that  the  protest  will  always  be  caused  by  an  alleged  impropiety  by  the  agency 
or  a  lack  of  full  communication  with  the  contractor.  The  fact  remains  that  a  number  of 
disappointed  bidders  will  file  a  protest  based  on  conditions  outside  of  the  control  of  the 
agency.  Laws  and  regulations  imposed  upon  the  agency  have  made  a  legitimate  effort  to 
improve  the  relationship  with  federal  contractors  by  streamlining  the  protest  process  and 
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enhancing  competition.  The  issue  arises:  have  the  lawmakers  reached  a  point  of  “leaving 
well  enough  alone  with  the  bid  protest  process?”  (Cantor,  1997:  155). 

Cantor  contends  that  several  recent  reforms  streamlined  the  bid  protest  system  by 
clarifying  jurisdictions,  discouraging  frivolous  protests,  and  eliminating  certain  forums 
(i.e.  GSBCA)  (Cantor,  1997:  156).  Former  Office  of  Procurement  Policy  Administrator 
Steve  Kelman  asserted  that  these  reforms  were  part  of  a  larger  effort  by  the  Government 
to  move  toward  “value  oriented”  and  “streamlined  competion”  (H.R  1670  Hearings). 
Cantor  acknowledges  these  goals  as  legitimate;  however,  he  points  to  a  shift  from 
fairness  to  efficiency  (Cantor,  1997:  157).  CICA  is  viewed  as  a  step  toward  full  and  open 
competition  to  promote  fairness;  however,  FASA  and  other  reforms  of  the  1990s  are  seen 
as  restrictive  to  the  process  (Cantor,  1997:  157).  Cantor  is  concerned  that  a  “sanctioning 
of  frivolous  protests”  and  a  consolidation  of  bid  protest  forums  will  discourage  a 
disappointed  bidder  from  filing  a  protest.  This  research  will  attempt  to  discern  a 
correlation  between  the  changes  in  protests  resulting  from  CICA  as  compared  to  the 
reforms  of  the  1990s. 

Previous  Research 

Lieberman  conducted  a  study  of  protests  filed  in  different  forums  over  a  three- 
year  period  and  classified  the  following  three  major  categories:  (1)  improper  actions  in 
evaluation  and  negotiation  of  proposals,  (2)  improper  actions  involving  solicitations  or 
requirements,  and  (3)  improper  restrictions  on  competition  (Lieberman,  1995:  11-19). 
These  official  reasons  were  labeled  the  “top  three  reasons”;  however,  no  evidence  is 
given  regarding  probable  causes  of  the  trends. 
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Cantor  presented  the  results  of  the  American  Bar  Association’s  Acquisition  Law 
Advisory  Panel  assembled  to  evaluate  bid  protests  related  to  procurement  reform.  The 
committee  recommended  the  consolidation  of  judicial  protests  and  recognized  the 
sanctioning  of  frivolous  protests  (Cantor,  1997: 170).  Cantor  warns  that  monetary 
sanctions  and  streamlining  the  bid  protest  process  in  the  interest  of  efficiency  have  a 
chilling  effect  on  the  amount  of  protests  filed  (Cantor,  1997:  1 76).  He  predicts  a 
reduction  of  protests  because  of  the  acquisition  reform  of  the  1990s;  however,  there  is  a 
lack  of  empirical  evidence  in  the  study.  Cantor’s  conclusions  were  based  upon  a  review 
of  court  cases  and  the  history  of  the  bid  protest  mechanism. 

Two  thesis  efforts  at  the  Air  Force  Institute  of  Technology  in  1997  analyzed  the 
effect  of  the  impact  of  the  Federal  Acquisition  Streamlining  Act  of  1994  on  post-award 
protest  frequency.  Duncan  concluded  that  that  FAS  A  resulted  in  a  marginal  decrease  in 
the  frequency  of  award  protests  (Duncan,  1997:  80).  While  Duncan  describes  the 
complex  dynamics  involved  in  this  adversarial  arena,  there  is  a  clear  need  to  search  for 
other  factors.  The  effect  of  improved  processes  or  new  acquisition  laws  on  the  number  of 
protests  filed  may  be  negligible  in  the  face  of  such  external  factors  as  the  size  of  the  DOD 
budget  and  the  general  economic  conditions. 

Thomas  focused  on  whether  FASA  changes  in  the  post-award  debriefing  process 
resulted  in  fewer  protests.  His  research  did  not  involve  quantitative  analysis  as  he 
recommended  the  research  to  “serve  as  the  basis  for  a  future  researcher  to  develop  a 
quantitative  study  on  the  effectiveness  of  the  post-award  debriefing  process”  (Thomas, 
1997:  90).  He  concluded  that  FASA  improved  the  post-award  debriefing  process  as  less 
adversarial  and  more  open  based  on  interviews/surveys-  He  continued  with  this  point  in 
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stating  that  this  atmosphere  may  reduce  the  likelihood  that  the  unsuccessful  offeror  will 
file  a  protest.  In  summary,  Thomas  advocated  the  use  of  a  statistical  analysis  with  a 
larger  sample  to  determine  the  impact  of  FAS  A  changes  on  the  post-award  debriefing 
process  on  protest  frequency. 

Mintz  used  multiple  regression  techniques  to  measure  a  variety  of  political  and 
economic  variables  on  the  number  of  contract  awards  per  year  (Mintz,  1 992: 22).  The 
statistical  analysis  provides  a  basis  for  measuring  factors  that  may  influence  the  number 
of  contracts  awarded  during  a  year.  In  the  same  manner,  this  research  will  attempt  to  use 
multiple  regression  techniques  to  quantify  the  effects  of  a  variety  of  independent 
variables  on  AF  GAO-Protests  Processed.  Throughout  the  course  of  the  literature  review, 
the  researcher  did  not  uncover  a  multiple  regression  model  to  gauge  contractor  protests. 
On  the  other  hand,  a  valuable  multiple  regression  model  (espoused  by  Mintz)  can  serve 
as  the  foundation  for  the  methodology  used  in  this  thesis. 

Other  Factors  Considered:  Contract  Dollars  and  Actions 

Throughout  the  history  of  the  United  States,  defense  expenditures  fluctuate  in 
relation  to  times  of  crisis  or  apparent  crisis  (Gansler,  1980:  9).  As  funds  flow  in  and  out 
of  the  defense  budget  based  on  public  opinion  and  world  events,  the  level  of  defense 
procurement  follows  an  extremely  cyclical  nature  (Gansler,  1980:  9).  Gansler  analyzed 
the  peaks  and  valleys  of  defense  spending  in  the  post  World  War  II  period.  He  concluded 
the  erratic  defense  spending  could  be  attributed  to  a  sense  of  constant  crisis  or  apparent 
crisis  in  the  United  States. 


19 


After  being  cut  by  almost  40%  from  1968-1977,  defense  spending  began  to  rise 
by  a  modest  $10  billion  in  1978  (OMB,  1991:  17).  The  historical  perspective  is  valuable 
in  understanding  the  nature  of  defense  spending;  however,  this  research  will  focus  on  the 
period  1984-1998.  Between  1980-1988  (known  as  the  Reagan  “spending  boom”),  real 
outlays  were  increased  by  about  $88  billion  (54%)  (OMB,  1991: 17).  A  series  of 
stunning  changes  in  the  world  during  1991  led  to  a  sharp  reduction  in  defense  spending 
(Lall  and  Marlin,  1992:  9). 

Given  the  huge  military  buildup  of  the  1980s,  a  significant  number  of  layoffs 
among  major  defense  contractors  began  occurring  in  1989.  McDonnell  Douglas  began  to 
retrench  in  1989  and  announced  plans  to  layoff  10,000  to  15,000  workers  (Lall  and 
Tepper,  1992:  9).  In  the  era  of  a  leaner  DOD  budget,  could  thicker  competition  and  a 
quest  for  survival  lead  to  an  increased  amount  of  protests?  Alternatively,  would  the 
effect  on  the  number  of  protests  be  negligible  as  the  contractors  avoided  fostering  ill  will 
in  the  tough  times,  sought  business  in  the  private  sector,  or  consolidated  among 
themselves? 

GAO  Sustain  Rate 

In  a  1995  study  conducted  by  Lieberman,  the  results  of  protests  filed  in  five 
different  forums  were  compared  to  identify  where  a  contractor  would  tend  to  have  their 
protest  sustained.  The  results  indicated  that  a  contractor  would  be  two  to  three  times 
more  likely  to  obtain  relief  at  the  General  Services  Board  of  Contract  Appeals  (GSBCA) 
(Lieberman,  1995:  2).  Overall,  the  GAO  averaged  around  12-13  percent  sustain  versus 
close  to  40  percent  for  the  GSBCA.  Lieberman  concludes  that  the  substantial  difference 
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in  sustain  rate  in  the  different  forums  will  be  significant  to  contractors  and  government 
officials  in  light  of  the  elimination  of  the  GSBCA.  Consequently,  the  overall  lower 
sustain  rate  at  the  GAO  might  reduce  the  number  of  protests  with  the  elimination  of  the 
GSBCA  as  a  viable  option. 

Figures  released  by  the  GAO  lend  credence  to  Lieberman’s  study.  The  GAO’s 
sustain  rate  has  remained  relatively  constant  throughout  the  1990s  between  11%  and  13% 
(Gordon,  1998:  5).  In  addition,  statistics  released  in  early  1998  reveal  a  decline  in  bid 
protests  by  18%  below  the  previous  year.  Why  do  the  protests  appear  to  be  falling?  The 
Government  Contractor  contends  that  this  could  be  the  result  of  contractor  consolidation, 
fewer  contracts,  fewer  contract  dollars,  and  the  changes  brought  about  by  FASA 
(Government  Contractor,  1998:  42). 

At  the  Air  Force  level,  the  number  of  GAO  protests  has  declined  from  a  high  of 
493  in  1993  to  203  in  1998  (SAF/AQC).  This  reduction  in  the  number  of  protests  filed 
must  be  viewed  in  light  of  the  sustain  rate.  While  the  protests  filed  have  declined 
dramatically,  the  AF-GAO  sustain  rate  has  declined  to  around  2%  in  the  late  1990s.  Are 
the  protest  numbers  falling  because  of  less  chance  for  relief  at  the  GAO  as  Lieberman 
predicted?  This  research  will  use  statistical  tools  in  an  attempt  to  answer  these  questions. 

General  Economic  Conditions:  Unemployment  Rate 

At  the  foundation  of  the  impact  of  general  economic  conditions  is  the  basic 
question:  will  economic  “bad  times”  increase  the  likelihood  of  contractor  protest? 
Classifying  general  economic  conditions  in  terms  of  inflation,  GNP,  unemployment,  etc. 
is  a  matter  of  gathering  U.S.  economic  data.  On  the  other  hand,  selecting  a 
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macroeconomic  variable  applicable  to  this  study  requires  further  analysis.  Mintz 
expanded  on  this  problem  in  constructing  his  regression  model  when  he  concluded 
“specifying  the  appropriate  economic  trigger  is  difficult,  particularly  since  the  defense 
budget  creates  substantial  economic  effects  of  its  own. . .  [there  are]  a  large  numbe  of 
economic  variables”  (Mintz,  1992:  22). 

Mintz  selected  the  unemployment  rate  as  a  quantitative  variable  in  the  statistical 
model  of  contract  awards.  When  political  leaders  speak  of  the  economic  benefits  of 
defense  spending,  there  is  a  good  chance  they  mean  jobs  in  defense  industry,  created  by 
large  military  installations  and  the  like  (Mintz,  1992: 22).  Consequently,  he  relates  the 
level  of  unemployment  to  public  attitudes  toward  the  economy  and  popularity  of  political 
candidates  (Mintz,  1992:  22). 

Beyond  public  attitudes  toward  the  economy,  high  unemployment  equates  to  low 
industrial  productivity  and  inefficient  use  of  industrial  and  social  capital  (Janorski,  1990: 
1).  Baumer  and  Van  Horn  label  the  unemployment  rate  as  the  “single  most  important 
signpost  of  a  nation’s  prosperity  due  to  the  costs  brought  about  by  the  reduced 
productivity”  (Baumer  and  Van  Horn,  1985: 1).  The  costs  of  unemployment  can  be 
quantified  by  realizing  that  a  1%  rise  in  the  unemployment  rate  deprives  the  federal 
treasury  of  approximately  $25  billion  in  lost  tax  revenues  and  increased  spending  on 
social  welfare  programs  (Baumer  and  Van  Horn,  1985:1).  Therefore,  the  nation’s 
unemployment  rate  will  provide  a  general  measurement  of  the  state  of  the  economy  for  a 
particular  year. 

This  study  will  not  attempt  to  define  what  level  of  unemployment  is  considered 
good  or  bad  because  the  answer  to  this  question  changes  over  time  based  on  ideology  and 
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partisanship  (Baumer  and  Van  Horn,  1985:  21).  During  the  1950s  and  1960s,  4%  was  the 
widely  accepted  measure  of  acceptable  unemployment  levels;  however,  economists 
argued  that  the  number  should  be  revised  to  6%  (Baumer  and  Van  Horn,  1985: 21).  This 
researcher  does  not  classify  the  unemployment  rate  against  a  baseline;  however,  an 
elevation  in  the  unemployment  rate  may  characterize  increased  negative  condition  in  the 
national  economy. 

Federal  contractors  at  the  operational  and  systems  level  conduct  business  in  the 
public  and  private  sector.  As  business  concerns,  they  are  exposed  to  conditions  in  the 
national  economy  and  make  adjustments  to  ensure  economic  survival.  During  periods  of 
negative  condition  in  the  economy  (higher  unemployment  periods),  federal  contractors 
might  behave  more  competitively  to  win  an  AF  contract.  In  strained  economic  times,  the 
disappointed  bidder  may  file  a  protest  with  less  provocation.  On  the  other  hand, 
perceived  “bad  times”  could  result  in  contractor  behavior  that  avoids  a  conflict  with  the 
AF.  Trends  or  shifts  in  the  protest  data  associated  with  general  economic  conditions 
(unemployment  rate)  will  be  tested  for  statistical  significance. 

Focus  of  Current  Study 

This  literature  review  confirms  a  lack  of  research  accomplished  to  date  that 
objectively  measures  the  effectiveness  of  acquisition  reform  and/or  external  factors  on 
the  number  of  protests  filed.  On  the  other  hand,  the  review  led  to  the  discovery  of  a 
political-economic  model  created  to  estimate  contract  awards  based  on  a  multitude  of 
independent  variables. 
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Mintz  used  a  political-economic  model  of  defense  spending  to  predict  contract 
awards  using  multiple  regression  techniques.  Using  a  variety  of  qualitative  and 
quantitative  variables  in  a  statistical  model,  he  demonstrated  how  a  major  portion  of 
defense  spending  (contract  awards)  fit  into  the  broader  picture  of  the  American  political 
economy  (Mintz,  1992: 29).  This  study  uses  elements  of  the  Mintz  model  as  a  basis  and 
justification  for  using  statistical  tools  in  the  research  design. 

This  research  attempts  to  determine  the  effects  of  acquisition  reform  initiatives, 
GAO  sustain  rate,  contract  dollars,  contract  actions,  and  general  economic  condition  on 
the  number  of  protests  filed.  Regression  analysis  will  be  employed  to  test  the  statistical 
significance  of  the  multiple  regression  model’s  response  and  predictor  variable(s).  In 
order  to  make  an  accurate  assessment  of  acquisition  reform’s  effect  on  bid  protests,  the 
study  will  provide  an  objective  look  at  protest  trends  and  associated  conditions  to  the  Air 
Force.  A  historical  analysis  that  culminates  with  statistical  results  may  shed  light  on  the 
utility  of  continued  reforms  in  the  bid  protest  arena.  While  the  results  of  the  research  will 
not  establish  causal  relationships,  the  statistical  correlation  provides  Air  Force  decision¬ 
makers  a  tool  to  identify  trends  in  contract  protest  frequency. 

Summary 

This  chapter  provided  a  review  of  literature  on  the  bid  protest  process,  reform  of 
acquisition  laws  as  related  to  bid  protests,  costs  associated  with  protests,  and  other  factors 
that  may  influence  the  number  of  protests  filed.  In  addition,  it  provided  relevant 
background  information  to  frame  the  study  in  the  appropriate  context  against  other 
research  in  the  area.  The  protest  process  and  key  terms  were  described  in  detail  with 
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current  trends  in  the  numbers  of  protests  filed.  Changes  in  procedure  and  substance 
brought  about  by  CICA  and  FASA  were  discussed  as  related  to  the  current  study.  The 
discourse  provided  insight  on  other  factors  besides  the  acquisition  reform  initiatives  that 
may  influence  the  number  of  protests  filed.  The  review  brought  the  complexity  of  the 
decision  to  file  a  protest  to  light  and  its  importance  due  to  the  nonmonetary  and  monetary 
costs  associated  with  a  protest. 

Previous  research  has  focused  on  the  reasons  that  contractors  file  protest  as 
opposed  to  the  environmental  factors  that  may  influence  their  decision.  While  some 
qualitative  approaches  have  been  made  to  discern  the  effects  of  FASA  and  post-award 
debriefings,  a  broad  statistical  analysis  has  not  been  conducted  in  the  Air  Force  regarding 
the  other  factors  brought  to  light  in  the  literature  review.  This  study  will  use  archival 
data  and  investigate  changes  in  the  numbers  of  protests  filed  over  a  14-year  period  to 
discover  statistical  significance.  Due  to  the  passage  of  CICA  and  significant  changes  it 
imposed  on  the  bid  protest  system,  1984  has  been  chosen  as  starting  point  for  the 
analysis.  The  GAO  forum  has  been  selected  as  the  point  of  investigation  because  of  the 
vast  amount  of  data  maintained  by  S  AF/AQC  regarding  this  entity  and  overwhelming 
choice  by  contractors  to  use  this  mechanism. 

How  have  the  acquisition  reforms  affected  the  number  of  AF  GAO-Protests 
Processed?  Have  the  number  of  contract  dollars  and  actions  affected  these  numbers  in 
conjunction  with  the  reforms?  Does  the  GAO  sustain  rate  have  a  statistical  impact  on  the 
numbers  of  protests  filed  in  the  AF?  Does  general  economic  condition  (i.e. 
unemployment  rate)  have  an  impact  on  the  numbers  of  protests  filed  in  the  AF?  The 


25 


existing  research  has  not  adequately  addressed  these  questions.  The  next  chapter  focuses 
on  the  research  methodology  employed  to  address  these  issues. 
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III.  Methodology 

Introduction 

This  chapter  describes  the  research  procedures  followed  to  meet  the  objectives 
outlined  in  Chapter  1 .  A  discussion  of  the  qualitative  and  quantitative  research 
paradigms  concludes  with  a  justification  of  the  method  selected  for  the  current  study. 

The  hypotheses  tested  and  ex  post  facto  independent  variables  are  explained.  Next,  the 
study  lists  the  benefits  and  shortcomings  of  the  Mintz  multiple  regression  model  adapted 
for  use.  The  time-series  analysis  is  introduced  as  a  potential  method  to  control  for  the 
pitfalls  of  internal  validity  associated  with  multiple  regression  in  this  study.  The 
discussion  of  the  time-series  analysis  and  regression  analysis  concludes  with  the  selection 
of  the  appropriate  method  for  the  study.  Lastly,  the  researcher  provides  the  method  for 
managing  and  recording  the  data. 

Quantitative  and  Qualitative  Research 

The  fundamental  approach  to  answer  the  research  question  is  grounded  in  the  two 
paradigms — quantitative  and  qualitative.  Beyond  the  philosophical  base  of  the  research, 
the  paradigms  have  an  intrinsic  link  to  the  type  of  methodology  selected  for  the  study 
(Cook  and  Reichardt,  1979: 11).  The  paradigms  exist  on  a  continuum  that  overlaps  in 
certain  realms  to  aid  the  researcher  in  the  pursuit  of  knowledge  in  a  scientific  endeavor. 

Firestone  classifies  the  quantitative  and  qualitative  paradigms  in  four  areas: 
assumptions,  purpose,  approach,  and  researcher  role.  The  quantitative  approach  seeks 
out  an  objective  reality  with  facts;  however,  the  qualitative  approach  sees  a  socially 
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constructed  reality.  While  the  quantitative  researcher  attempts  to  discover  causes  and 
relationships,  the  qualitative  researcher  seeks  understanding.  Quantitative  research  tends 
to  be  characterized  as  experimental  and  correlational;  whereas,  qualitative  research  is 
often  labeled  as  a  form  of  ethnography.  In  summary,  the  qualitative  research  approach  is 
used  when  observing  and  interpreting  reality  with  the  aim  of  developing  a  theory  that  will 
explain  the  phenomena.  The  quantitative  approach  begins  with  a  theory/hypothesis  and 
“tests  for  confirmation  or  disconfirmation  based  on  the  data”  (Newman  and  Benz,  1998: 
2-3). 

While  theoretical  debate  continues  about  the  paradigms  and  associated  methods, 

practical  research  recommends  a  decision  based  on  the  “best  or  most  logical  link  between 

the  paradigm  and  method”  (Cook  and  Reichardt,  1979: 17).  A  guide  in  selecting  the 

appropriate  method  is  best  summarized  as: 

Researchers  who  use  the  qualitative  methods  do  subscribe  to  the  qualitative 
paradigm  than  to  the  quantitative  paradigm.  Similarly,  there  is  a  correlation 
between  the  use  of  quantitative  methods  and  adherence  to  the  quantitative 
paradigm  (although  these  linkages  between  paradigm  and  method  are  not  always 
perfect,  as  many  seem  to  believe).  (Cook  and  Reichardt,  1979:  17) 

Cook  and  Reichardt  conclude  that  the  most  important  distinction  to  aid  in  the  selection  of 
a  paradigm  and  method  is  the  dimension  of  verification  versus  discovery.  Quantitative 
methods  verify  or  confirm  theories/hypotheses  and  qualitative  methods  seek  to  discover 
or  generate  theories. 

The  specific  methods  associated  with  the  qualitative  paradigm  fall  under  the 
heading  of  ethnography.  They  consist  of  case  studies,  field  studies,  grounded  theory,  and 
descriptive  studies  (Creswell,  1994:  11).  Quantitative  methods,  characterized  as 
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empirical  or  statistical,  include  experiments,  surveys,  and  generalizing  from  a  sample  to  a 
population  (Creswell,  1994:  11). 

The  respective  paradigms  coexist  in  a  complimentary  relationship  with  each 
other.  Newman  and  Benz  contend  that  the  place  of  theory  overlaps  in  the  world  of  the 
quantitative  and  qualitative  researcher.  The  qualitative  researcher  is  motivated  by 
“theory  building”  and  the  quantitative  researcher  is  motivated  by  “theory  testing;” 
however,  each  depends  on  the  other  to  cover  the  research  realm  (Newman  and  Benz, 

1998:  20).  For  clarification.  Figure  1  has  been  inserted  on  the  following  page  adapted 
from  Qualitative-Quantitative  Research  Methodology.  Figure  1  shows  the  location  of  the 
“theory”  at  neither  the  beginning  nor  the  end  of  the  Newman-Benz  model.  The  circles 
represent  the  qualitative  researcher  and  the  squares  symbolize  the  quantitative  researcher 
on  the  model.  The  squares  within  the  circles  illustrate  the  overlap  between  the  two 
paradigms  that  characterize  the  complimentary  relationship. 

Following  the  cycle  in  Figure  1,  the  qualitative  researcher  approaches  the  topic 
with  an  inductive  approach.  Data  is  gathered  regarding  the  item  to  be  measured  (Circle 
A)  using  the  methods  described  earlier.  The  researcher  continues  through  the  A-B-C-D- 
E  process  in  an  effort  to  produce  a  theory  (Circle  E).  The  quantitative  researcher  begins 
with  the  Theory  (Square  1)  and  deductively  follows  the  1-2-3-4-5-6  path  in  the  figure  to 
derive  conclusions  and  recommendations.  The  dotted-lined  box  surrounding  Theory  1 
and  Theory  E  represents  the  point  at  which  the  quantitative-qualitative  gap  is  closed  and 
the  cycle  is  complete.  Newman  and  Benz  formulated  this  model  to  illustrate  the  value  of 
a  holistic  view  of  the  two  paradigms  and  their  value  to  the  researcher. 
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Figure  1.  The  Qualitative-Quantitative  Philosophy  of 
Educational  Research  (Newman  and  Benz,  1998) 


The  Appropriate  Method 

In  selecting  the  appropriate  method  for  the  study,  the  researcher  must  always 
consider  the  ultimate  goal:  the  most  effective  way  to  search  for  knowledge  (the  “truth”) 
(Newman  and  Benz,  1998:  1 1).  Instead  of  advocating  a  particular  paradigm  as  a 
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universal  approach,  the  selection  must  serve  to  answer  the  research  question.  On  a 
practical  level,  the  researcher  must  consider:  (1)  the  nature  of  the  research  question,  (2) 
accessibility  of  the  data,  and  (3)  whether  the  data  are  quantified  according  to  the  design 
of  the  study. 

In  this  study,  the  nature  of  the  research  question  and  background  information  lend 
itself  to  quantitative  methods.  The  research  seeks  to  discover  correlation  and  statistical 
relationships  between  the  number  of  protests  filed  (dependent  variable)  and  acquisition 
reform  initiatives,  GAO  sustain  rate,  contract  dollars  and  actions,  and  general  economic 
condition  (all  independent  variables).  The  literature  review  presents  a  number  of  theories 
espoused  by  previous  research  and  experts  in  the  acquisition  field  on  the  research 
question. 

Secondly,  archival  data  exists  at  SAF/AQCX  to  track  the  relationship  over  the 
relevant  14-year  period  of  the  study.  In  this  study,  there  is  no  need  to  employ 
ethnographic  methods  associated  with  the  qualitative  paradigm.  In  addition,  the 
researcher  is  detached  from  the  data  to  compose  an  objective  analysis  based  on  statistical 
procedures  discussed  later  in  this  section. 

The  data  collected  from  SAF/AQCX  has  been  quantified  according  to  the 
research  design.  In  addition,  the  study  assigns  a  number  to  the  variables  for  the 
application  of  statistical  methods. 

Selection  of  Quantitative  Research  Design.  After  an  analysis  of  both  paradigms 
and  associated  methods,  a  quantitative  research  design  is  most  appropriate  to  address  the 
research  question.  In  this  study,  reality  is  singular  and  apart  from  the  researcher  as 
required  by  the  quantitative  paradigm  (Creswell,  1994:  5).  As  archival  data  analysis  and 
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statistical  techniques  are  used,  the  researcher  is  independent  of  the  number  of  protests 
filed  (dependent  variable).  Every  effort  has  been  made  to  ensure  that  all  research  will  be 
conducted  free  of  bias  and  value  consistent  with  the  quantitative  view  (Creswell,  1994: 
5).  The  goal  of  the  research  is  to  determine  the  existence  of  a  relationship/correlation 
between  the  dependent  and  independent  variables  for  the  use  of  AF  decision-makers. 
Conclusions  and  recommendations  exhibit  the  appropriate  degree  of  tentativeness  and 
generality  for  the  study. 

Propositions  and  Hypotheses 

The  following  propositions  and  hypotheses  are  proposed  for  the  study: 

Proposition#!.  Chapter  II  discusses  the  peaks  and  valleys  in  defense  spending 
associated  with  the  post  World  War  II  period  and  how  procurement  follows  the  cyclical 
nature  of  the  DoD  budget.  In  this  manner,  the  1980s  saw  an  increase  in  contract 
awards/dollars  and  the  1990s  saw  a  dramatic  decline  in  these  variables.  In  a  lean  defense 
contracting  environment,  the  issue  arises  whether  or  not  the  contractor  would  be  more 
likely  to  fight  for  the  federal  dollar.  In  other  words,  does  vying  for  a  smaller  pool  of  DoD 
dollars  lead  to  an  increased  number  of  protests?  On  the  other  hand,  a  protest  may  foster 
ill-will  between  a  contractor  and  AF  Agency.  Would  a  contractor  be  willing  to  risk 
deteriorated  relations  in  a  time  of  defense  cutbacks  of  federal  contracting  dollars?  This 
study  contends  that  an  increase  in  AF  Contract  Dollars  and  AF  Contract  Actions  may 
actually  decrease  the  number  of  protests  as  more  contracts  would  be  awarded. 
Consequently,  the  larger  pool  of  contracts  would  quell  the  tendency  to  file  protests.  The 
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number  of  AF  contract  actions  and  contract  dollars  have  impacted  the  AF-GAO 
protests  filed  in  a  manner  that  would  significantly  alter  the  number  of  protests. 

Hp  An  increase  in  the  number  of  AF  contract  actions  will  decrease  the  number 
AF  GAO-Protests  Processed  from  1985-1998. 

H?:  An  increase  in  the  AF  contract  dollars  will  lead  to  a  decrease  in  the  amount  of 
AF  GAO-Protests  Processed. 

Proposition  #2.  Chapter  II  introduces  Lieberman’s  comparison  of  protests  filed  in 
five  different  forums  to  identify  where  a  contractor  would  most  likely  have  their  protest 
sustained.  Lieberman’s  study  points  this  research  to  an  investigation  of  the  relationship 
between  the  GAO  sustain  rate  and  the  number  of  protests  filed  in  this  forum.  On  the 
surface,  it  appears  that  the  GAO  sustain  rate  has  remained  higher  during  periods  of 
increased  protest  frequency  and  lower  during  periods  of  reduced  protests  frequency.  The 
GAO  sustain  rate  has  influenced  the  AF-GAO  protests  filed  in  a  manner  that  would 
significantly  alter  the  number  of  protests. 

H3:  An  increase  in  AF-GAO  sustain  rate  can  be  associated  with  an  increase  in  the 
number  AF  GAO-Protests  Processed 

Proposition  #3.  This  proposition  addresses  the  following  question:  will  economic 
bad  times  increase  the  likelihood  of  contractor  protest?  Chapter  II  discusses  the  Mintz 
model  in  relation  to  this  variable  and  justified  the  selection  of  the  unemployment  rate  as 
an  adequate  barometer  of  the  economy  for  a  period.  This  study  contends  that  periods  of 
relative  prosperity  and  good  times  in  the  economy  (signified  by  a  decrease  in  the 
unemployment  rate)  will  be  associated  with  fewer  protests.  Conversely,  a  downturn  in 
the  economy  (signified  by  an  increase  in  the  unemployment  rate)  will  bring  about  an 
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increase  in  the  number  of  protests.  The  perceived  bad  times  may  cause  the  contractor  to 
battle  for  economic  survival  in  the  commercial  and  public  sectors  of  the  economy.  The 

unemployment  rate  (General  Economic  condition)  has  impacted  the  AF-GAO 
protests  filed  in  a  manner  that  would  significantly  alter  the  number  of  protests. 

H4:  Periods  of  higher  unemployment  can  be  associated  with  increases  in  the 
number  of  AF  GAO-Protests  Processed. 

Proposition  #4.  Various  acquisition  reform  initiatives  have  sought  to  streamline 
the  procurement  system,  maintain  fairness  in  the  competitive  process,  and  improve 
Government-Contractor  relations.  Procurement  officials  believe  that  the  push  for  fairness 
and  competition  improvements  will  lead  to  a  reduction  in  contractor  protests.  There  has 
been  a  reduction  in  the  number  of  AF-GAO  protests  filed  during  the  late  1990s;  however, 
the  relationship  to  acquisition  reform  is  largely  untested.  This  researcher  asserts  that 
acquisition  reform  initiatives  have  not  affected  the  AF-GAO  protests  filed  in  a 
manner  that  would  significantly  alter  the  number  of  protests. 

H5:  The  introduction  of  FASA  and  FAR  15  rewrite  (efficiency  changes)  did  not 
reduce  the  mean  number  of  AF  GAO-Protests  Processed. 

Sources  of  Data  Analysis 

The  GAO-Protests  Processed  in  the  Air  Force  has  been  obtained  for  the  period 
between  1984-1998.  SAF/AQCX  maintains  electronic  and  paper  data  bases  to  track  the 
number  of  protest  filed  in  the  Air  Force  at  the  GAO,  the  number  of  contract  actions  for 
the  fiscal  year,  the  number  of  contract  dollars  for  the  fiscal  year  and  the  GAO  sustain  rate 
of  AF  protests.  The  significant  acquisition  reform  variables,  classified  by  CICA  (’84), 
FASA  (’94),  and  FAR  15  rewrite  (’97)  are  verified  for  coding  through  secondary  data 
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sources.  In  addition,  the  study  gauges  general  economic  condition  by  the  U.S 
unemployment  rate. 

Due  to  the  nature  of  the  independent  variables,  they  are  characterized  as  non- 
manipulatable.  Since  the  independent  variables  are  out  of  the  control  of  the  researcher, 
the  research  can  be  classified  as  ex  post  facto  or  correlational  research  (Newman  and 
Benz,  1998: 41).  Research  design  experts  warn  against  the  possible  misinterpretation  of 
research  where  there  is  no  control  over  the  independent  variables.  As  this  study  is 
correlational  research,  the  researcher  and  subsequent  user  of  the  results  must  be  careful  to 
remember  that  “correlation  is  not  necessarily  causation”  (Newman  and  Benz,  1998: 41). 
In  this  thesis,  one  must  take  care  in  assuming  that  acquisition  reform  caused  a  reduction 
of  protests.  This  is  precisely  why  a  number  of  variables  have  been  introduced  for  study 
and  comparison.  In  order  to  assume  a  causal  relationship,  the  researcher  must  have 
internal  validity  with  all  other  possible  explanations  for  the  effect  on  the  dependent 
variable  controlled. 

The  researcher  acknowledges  the  lack  of  control  in  the  ex  post  facto  design; 
however,  attempts  have  been  made  to  address  issues  of  the  internal  validity  of  the  study. 
The  independent  variables  cannot  be  manipulated  because  they  have  already  occurred; 
however,  the  data  can  be  analyzed  over  a  time  series  to  discover  the  statistical  impact  of 
the  independent  variable  with  respect  to  pre-  and  post-  events.  To  address  the  self¬ 
selection  of  variables  confound,  the  researcher  selects  the  variables  and  derives  the 
hypotheses  from  the  existing  body  of  qualitative  research  in  the  protest  area.  Every  item 
tested  in  this  study  has  been  proposed  and  developed  by  previous  researchers  and  not 
arbitrarily  selected  by  the  researcher. 
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In  summary,  ex  post  facto  research  is  not  appropriate  for  a  research  question 
dealing  with  causation.  Since  this  research  deals  with  relationships,  the  use  of  the  ex  post 
facto  design  is  appropriate  (Newman  and  Benz,  1998:  42). 

Benefits  and  Shortcomings  of  Using  a  Multiple  Regression  Model 

Instead  of  providing  a  mere  correlation  analysis,  regression  techniques  provide  an 
equation  describing  the  nature  of  the  relationship  between  the  variables  (Kachigan,  1986: 
238).  Secondly,  regression  analysis  produces  measures  of  variance  that  allow  the 
researcher  to  assess  the  power  of  the  chosen  model  for  estimation  and  prediction. 
(Kachigan,  1986:  238).  In  summary,  the  formulation  of  the  regression  equation  and 
subsequent  analysis  of  variance  elevate  it  above  a  “curve  fitting  technique”  (Kachigan, 
1986:  238). 

In  regression  analysis,  the  dependent  variable  is  always  quantitative,  but  the 
independent  variables  may  be  either  quantitative  or  qualitative.  The  way  in  which  an 
independent  variable  enters  the  model  depends  on  its  type  (nominal,  ordinal,  interval, 
ratio  data)  (McLave  and  Benson,  1998:  579).  Mintz  provides  a  useful  model  to  measure 
contract  awards  versus  a  variety  of  independent  variables  (Mintz,  1992:  23). 

This  study  applies  regression  analysis  in  the  same  manner  that  Mintz  used  the 
procedure  to  predict  the  number  of  contract  awards  in  a  particular  year.  The  researcher 
adapts  the  Mintz  Model  to  measure  the  number  of  AF  protests  filed  as  a  function  of  the 
independent  variables  in  the  study. 
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The  specifications  of  Mintz’s  statistical  model  adapted  for  the  model  of  AF  GAO- 
Protests  Processed  follow: 

y  =  Po  +  Pl(Xl)  +  P2(X2)  +  P3(X3)  +  P4(X4)  +  P5(X5)  +  € 

where  the  following  apply: 

y  =  AF  GAO-Protests  Processed  during  year  (y) 
xi  =  AF  Contract  Actions 
x2  =  AF  Contract  Dollars 
X3  =  AF-GAO  Sustain  Rate 

X4  =  U.S.  Unemployment  Rate  (General  Economic  Conditions) 

X5  =  Acquisition  Reform  FAS  A:  0  if  CICA  prior  to  FASA/FAR  15  rewrite 

1  if  after  FASA/FAR  15  rewrite 

When  conducting  a  multiple  regression  analysis,  threats  to  the  internal  validity  of 
the  study  surface.  McClave  and  Benson  warn  against  the  common  pitfalls  during  the 
testing  and  prediction  phase:  1)  parameter  estimability,  2)  multicollinearity,  and  3) 
prediction  outside  of  the  experimental  region  (McLave  and  Benson,  1998:  550-552). 
Concerning  parameter  estimability,  the  data  provides  more  than  three  independent  and 
dependent  variables  in  the  study.  Since  there  are  greater  than  three  data  points,  a  pattern 
for  regression  may  be  established  between  the  number  of  protests  filed  in  the  Air  Force 
and  the  independent  variables  over  a  14-year  period.  In  order  to  guard  against 
multicollinearity,  the  researcher  pays  special  attention  to  the  statistical  outputs  in  the 
regression  analysis.  If  an  independent  variable  has  a  VIF  score  above  100,  it  may  be 
overlapping/correlating  with  another  independent  variable.  Finally,  the  study  uses  the 
data  in  the  regression  analysis  to  make  estimations  within  the  relevant  region  (1984- 
1998).  If  the  model  is  extrapolated  beyond  this  region,  there  is  a  risk  of  grossly 
inaccurate  predictions. 
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The  Mintz  model  serves  a  useful  purpose  in  providing  a  statistical  foundation  to 
analyze  a  contracting  issue  in  terms  of  independent  variables;  therefore,  this  study  uses 
the  techniques  in  a  similar  manner.  On  the  other  hand,  there  are  threats  to  the  internal 
validity  of  the  model  that  must  be  addressed.  McLave  and  Benson  point  out  the  problem 
associated  with  using  a  regression  model  to  analyze  time  series  data  (McLave  and 
Benson,  1998:  553).  As  the  observations  occur  sequentially  in  this  study  over  a  period  of 
time  (1984-1998),  they  may  be  correlated  over  time.  In  addition,  the  assumption  of 
independent  errors  (fundamental  to  the  use  of  regression)  threatens  to  hamper  the  results 
if  violated  (McLave  and  Benson,  1998:  553).  To  overcome  these  problems,  McLave  and 
Benson  suggest  consideration  of  a  time  series  model  (McLave  and  Benson,  1998:  553). 
The  next  section  explores  the  possibility  of  using  the  time-series  model  for  this  study. 

Consideration  of  a  Time  Series  Model 

A  time  series  consists  of  observations  taken  sequentially  over  time  as  opposed  to 
the  standard  regression  model  where  the  order  is  irrelevant  (Pole  and  others,  1994:  3).  In 
this  study,  the  data  is  spaced  equally  by  the  year  associated  with  the  number  of  protests 
filed.  Pole  points  out  the  difference  between  time  series  analysis  and  forecasting  by 
establishing  “the  construction  of  a  suitable  model  based  upon  an  analysis  of  the  historical 
development  of  the  series  and  information  relevant  to  the  series’  likely  future 
development”  as  the  ultimate  goal  (Pole  and  others,  1994: 4).  Time-series  data 
eliminates  hypotheses  associated  with  a  particular  variable  (i.e.  GAO  sustain  rate,  general 
economic  condition);  however,  the  causal  connection  is  never  demonstrated  (Gottman, 
1981:45). 
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To  describe  the  variation  in  the  number  of  protests  filed  over  the  14  year  time 
series,  the  quantitative  researcher  might  construct  a  model  for  the  process  that  may  have 
generated  the  data  set.  Time-domain  models  or  frequency-domain  models  are  available 
to  apply  this  method  to  analyze  the  variation  associated  with  the  data  (Gottman,  1981 : 
46).  In  this  theoretical  example,  a  variation  in  two  sets  of  time-series  sets  could  be 
established;  consequently,  one  time-series  may  be  used  to  forecast  the  other. 

In  addition,  the  quantitative  researcher  may  employ  the  use  of  an  interrupted  time- 
series  design  to  discern  the  effects  of  acquisition  reform  and  general  economic  condition. 
In  the  study,  the  number  of  protests  is  the  dependent  variable  being  studied  over  time 
(yt).  The  intervention  variables  (xj  and  X2)  represent  the  changes  imposed  on  the  time 
series  (interruptions).  The  intervention  of  acquisition  reform  and  the  general  economic 
conditions  could  be  modeled  and  analyzed  for  a  relationship  in  this  stochastic  model. 

On  the  surface,  the  time-series  data  analysis  appears  to  yield  the  most  accuracy 
and  validity;  however,  additional  investigation  uncovers  the  obstacles  that  block  its 
implementation  in  this  thesis.  Gottman  issues  a  warning  about  the  practical  issue  of 
power  in  a  time  series  analysis: 

If  a  graduate  student  were  to  propose  a  dissertation  employing  a  two-group 
analysis  of  variance,  but  said  that  he  or  she  planned  to  use  only  three  subjects  in 
each  cell,  there  had  better  be  a  good  reason  for  doing  that,  because  such  a  study 
would  have  little  power  to  detect  differences  between  the  means.  (Gottman,  1981 : 
58) 

Gottman  acknowledges  that  time-series  analysis  is  possible  with  few  data  points; 
however,  he  describes  this  undertaking  as  “risky  business”  that  leads  to  insignificant 
results  based  on  insufficient  data  (Gottman,  1981 :  59).  In  summary,  the  time-series 
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analysis  requires  a  greater  number  of  data  points  to  fit  a  model  than  are  available  for  this 
research. 

Kendall  cites  examples  of  an  appropriate  time-series  design:  analysis  of  yields  of 
barley  over  a  55  year  period  and  miles  flown  by  an  airline  for  96  months  (Kendall,  1973: 
9).  In  both  examples,  the  time  of  observation  and  period  covered  would  provide 
sufficient  data  points  to  establish  trends/long  term  movement,  fluctuations  about  the 
trend,  and  a  seasonal  component.  In  this  study,  the  number  of  protests  collected  over  a 
14-year  period  do  not  provide  an  adequate  number  to  provide  meaningful  results  using 
the  time-series  analysis. 

Selected  Approach 

After  a  careful  comparison  of  both  procedures,  the  multiple  regression  approach 
meets  the  objectives  and  conditions  for  this  study.  The  use  of  this  method  best  suits  the 
problem  of  whether  AF  contract  protest  frequency  rises  or  falls  with  the  independent 
variables.  A  discussion  of  the  specific  method  of  data  collection  and  procedures  follows 
in  the  next  section. 

Managing  and  Recording  Data 

The  study  organizes  the  pertinent  data  collected  from  SAF/AQCX  and  secondary 
data  sources  with  an  organized  approach  advocated  by  Creswell.  A  guide  to  the  specific 
procedures  to  focus  the  data  collection  are  summarized  as  follows  (Creswell,  1994:  152): 

a)  Parameters :  AF  GAO  Protests  Processed  ‘84  -‘98,  AF  Contract 
Actions/Dollars  ‘84-’98,  U.S.  Unemployment  Rate  ’84  -  ’98; 
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b)  Information  Collected :  Number  of  GAO  Protests  Processed,  AF  Contracts 
Dollars,  AF  Contract  Actions,  and  U.S.  Unemployment  Rate; 

c)  Protocol :  Statistical  records  maintained  by  SAF/AQCX,  U.S.  Department  of 

Commerce. 

The  parameters  ensure  a  relevant  and  obtainable  sample  of  AF  contract  protests  for 
analysis  in  the  study.  Protests  awarded  during  and  after  1984  reflect  the  changes  imposed 
by  the  Competition  in  Contracting  Act  (CICA)  and  the  Federal  Acquisition  Streamlining 
Act  (FASA)  /  FAR  15  rewrite.  Before  the  1980s,  the  bid  protest  operated  under  different 
laws  and  regulations  changing  the  face  of  protests. 

The  data  are  presented  in  a  time  series  format  to  show  AF  contract  protest  trends 
in  relation  to  acquisition  reform,  the  size  of  the  defense  budget,  and  general  economic 
conditions.  Since  the  standardized  data  is  capable  of  being  compared  over  successive 
time  intervals,  a  longitudinal  study  will  be  undertaken  (Miller,  1991 : 21).  Changes  in  AF 
contract  protest  trends  may  or  may  not  be  correlated  to  changes  imposed  by  acquisition 
reform;  however,  there  is  a  combination  of  factors  contributing  to  the  phenomena.  There 
is  inherent  difficulty  in  looking  at  protests  strictly  in  terms  of  acquisition  reform. 
Contemporary  approaches  involve  allowing  for  and  expecting  a  number  of  different 
causes  for  a  single  event  (Miller,  1991 :  21).  Consequently,  the  justification  for  using 
multiple  regression  analysis  techniques  in  this  study  is  complete. 

Converting  Raw  Data  to  Variables.  A  dependent  variable  (y)  is  the  item  to  be 
modeled  and  the  independent  variables  (x)  attempt  to  predict  changes  in  (y).  In  this 
study,  the  dependent  variable  is  the  number  of  AF  GAO-Protests  Processed  per  year 
between  1984  and  1998.  The  independent  variables  are  AF  Contract  Actions,  AF 
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Contract  Dollars,  GAO  Sustain  Rate,  Unemployment  Rate  (General  Economic 
Conditions),  and  Acquisition  Reform  (CICA  or  FASA/FAR  1 5  rewrite  period).  An 
application  of  these  independent  variables  will  be  discussed  in  this  subsection. 

AF  Contract  Actions  consist  of  operations,  systems,  and  logistics  contracting 
actions  reported  to  SAF/AQC.  These  actions  could  be  formal  contract  awards  or 
modifications;  however,  both  expose  the  AF  to  potential  contractor  protest.  The  data  will 
be  measured  in  terms  of  millions  of  actions  and  gauge  the  level  of  activity  for  a  particular 
year. 

AF  Contracting  Dollars  consist  of  operations,  systems,  and  logistics  dollars 
reported  to  SAF/AQC  for  particular  year.  To  analyze  the  AF  Contracting  Dollars 
(measured  in  billions  of  dollars),  an  index  number  has  been  calculated  and  measured 
against  a  base.  An  index  number  is  a  number  that  measures  the  change  in  a  variable  over 
time  relative  to  its  value  during  a  specific  base  period  (McLave  and  Benson,  1998:  734). 
In  this  study,  1992  has  been  selected  to  describe  the  relative  changes  in  AF  Contracting 
Dollars  using  the  following  formulation  steps: 

1 )  1 9XX  Index  Number  =  19XX  Contract  Dollars  X  1 00 

1992  Contract  Dollars 

2)  1 9XX  Index  Number  X  1 9XX  Contract  Dollars  =  1 9XX  Contract  Dollars 

(converted) 

The  AF  GAO  Sustain  Rate  represents  the  percentage  of  protests  sustained  out  of 
the  total  protests  received  for  a  particular  year.  SAF/AQC  monitors  the  status  of  a  GAO 
protest;  therefore,  a  data  base  of  the  amount  of  sustained  protests  will  be  obtained  for 
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evaluation.  This  percentage  answers  the  question:  How  many  times  were  the  AF 
contractors  successful  at  the  GAO  forum? 

The  study  measures  general  economic  conditions  with  a  single  variable — the 
unemployment  rate.  The  selection  of  this  measure  of  the  general  economy  is  not  meant 
to  diminish  other  factors  in  macroeconomics  (inflation,  interest  rate,  GNP);  the  logic  and 
reasoning  behind  this  selection  was  outlined  in  Chapter  II.  The  unemployment  rate 
expresses  the  percent  of  the  United  States’  non-institutionalized  population  over  16  years 
old  that  is  not  employed  (GPO,  1992:  381).  The  study  derives  the  data  for  this 
independent  variable  from  the  U.S.  Department  of  Commerce  as  published  in  the 
Statistical  Abstract  of  the  United  States. 

To  measure  the  impact  of  acquisition  reform,  the  study  codes  variables  to 
represent  the  period  following  the  implementation  of  CICA  (1984  to  the  present)  and 
FAS  A/FAR  15  rewrite  (1994  to  the  present).  Chapter  II  describes  the  sweeping  reforms 
of  CICA  in  the  bid  protest  process;  therefore,  this  period  represented  increased  equity  in 
the  process.  FASA  and  the  FAR  15  rewrite  increased  the  level  of  judgment  and  business 
sense  required  in  a  streamlined  acquisition  process;  therefore,  they  will  be  characterized 
together.  Overall,  the  analysis  provides  a  comparison  between  the  perceived  equity 
associated  with  CICA  and  efficiency  associated  with  FASA/FAR  15  rewrite.  A 
discussion  of  the  coding  mechanisms  follows  in  the  next  subsection. 

Regression  Procedure:  Constructing  and  Testing  a  Model 

While  the  Mintz  model  provides  a  multiple  regression  model  with  qualitative  and 
quantitative  variables  to  measure  contract  awards,  the  researcher  cannot  simply  insert 


variables  for  this  study.  According  to  Mclave  and  Benson,  proper  model  building 
involves  developing  a  model  that  provides  a  “good  fit”  to  a  set  of  data  (McLave-Benson, 
1998:  578).  Most  importantly,  the  regression  model  must  be  built  using  a  methodical 
approach  to  “approximate  the  true  nature  of  the  relationship  between  the  mean  response 
E(Y)  and  the  independent  variables”  (McLave  and  Benson,  1998:  578).  Model  building 
serves  as  the  foundation  of  the  success  or  failure  of  the  regression  analysis;  therefore, 
much  care  is  given  to  this  area  of  the  regression  analysis.  Figure  2  illustrates  this  study’s 
approach  to  analyzing  data,  formulating  a  model,  testing  hypotheses,  and  obtaining 
results. 

In  Figure  2,  the  procedure  begins  by  gathering  the  data  in  accordance  with  the 
Managing/Recording  Data  section  of  this  chapter.  Next,  the  researcher  begins  with  a 
fundamental  issue  in  data  analysis:  what  types  of  variables  are  being  analyzed?  The 
classification  of  variables  as  quantitative  or  qualitative  defines  the  limitations  of  their  use 
and  application  to  any  statistical  analysis.  While  the  dependent  variable  will  be 
quantitative,  the  independent  variables  may  be  either  quantitative  or  qualitative  (McLave 
and  Benson,  1998:  579).  Quantitative  data  can  be  measured  on  a  naturally  occurring 
numerical  scale  and  qualitative  data  can  only  be  classified  into  categories.  Quantitative 
data  contains  two  subclassifications  applicable  to  this  study:  1)  interval  data  -  measured 
on  a  scale,  origin  has  no  meaning,  and  2)  ratio  data  -  highest  level,  origin  (0)  is 
meaningful  (McLave  and  Benson,  1998:  12). 
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Gather  data  from  SAF/AQCX:  AF-GAQ 
Protests  Processed,  AF  Contract  Awards, 
AF  Contract  Dollars  (1984-1998) 


Analyze  data; 

-Descriptive  Statistics 
-Graphical  Analysis  (Scattergrams) 


Model  Formulation 


Hypothesize  deterministic  component  of 
model: 

Relate  mean  E(y)  to  independent  variables: 

-  AF  Contract  Awards  (quantitative) 

-  AF  Contract  Dollars  (quantitative) 

-  GAO  Sustain  Rate  (quantitative) 

■  Unemployment  Rate  (quantitative) 

Acquisition  Reform  (qualitative) _ 


Test  Multiple  Regression  Model. 

Use  the  V-Heuristic  Technique 
-State  assumptions  and  estimate  paramaters 
-Statistically  evaluate  usefulness  of  model 
-Formulate  estimate  of  standard  deviation 


Perform  Residual  Analysis. 

-  Check  model  assumptions 


Determine  usefulness  of  the  selected 

model:  estimation  and/or  prediction. 

-  Derive  least  squares  model 

-  Calculate  predicted  vs  actual  values 


Conduct  t-tests  on  betas. 


-  Determine  "specific"  contributions 
to  the  protest  frequency 

-  Refine/apply  appropriate  model 


Figure  2:  (Study  Procedure/Implementation 
Model  (Based  on  guidance  from  McLave  and  Benson 
Chapters  11  and  12, 1998). 


Qualitative  data  contains  subclassifications  applicable  to  this  study:  nominal/categorical 
data  -  code  numbers  that  cannot  be  ordered  or  added/subtracted  (McLave  and  Benson, 


1998: 14).  Chapter  IV  classifies  the  independent  variables  to  their  appropriate  level  for 
use  in  this  study. 

Following  this  classification,  the  researcher  uses  graphical  methods  to  describe, 
summarize,  and  investigate  patterns  in  the  data.  A  time  series  plot  for  AF  GAO-Protests 
Processed  (horizontal  axis)  over  the  years  1984-1998  (vertical  axis)  has  been  constructed. 
From  this  plot,  the  researcher  attempts  to  identify  visual  trends  and  movements  in  the 
dependent  variable  over  time.  While  this  may  provide  information  regarding  possible 
trends  in  the  data,  the  researcher  exercises  care  in  making  any  judgments.  The  numerical 
methods  employed  in  the  descriptive  statistic  analysis  are  limited  to  the  summary 
statistics  related  to  AF  GAO-Protests  Processed  between  1984-1998.  The  mean  and 
standard  deviation  provide  the  average  number  of  protests  and  variation  over  the  years. 

As  part  of  the  graphical  analysis  in  Figure  2,  the  study  constructs  a  series  of 
scattergrams  to  determine  the  plausibility  of  a  relationship  between  AF  GAO-Protests 
Processed  (y)  and  Contract  Awards  (xi),  Contract  Dollars  (x2),  the  GAO  Sustain  Rate 
(x3),  or  the  Unemployment  Rate  (X4)  (McLave  and  Benson,  1998:  434).  The  researcher 
determines  if  a  positive,  negative,  or  no  relationship  exists  between  the  dependent  and 
quantitative  independent  variables. 

Following  the  graphical  analysis,  the  researcher  codes  the  qualitative  variable 
(acquisition  reform)  for  use  in  the  multiple  regression  model.  With  regard  to  the 
qualitative  variable  (acquisition  reform),  the  study  codes  the  data  as  appropriate  in  a 
tabular  format.  The  CICA  years  prior  to  FASA/FARA  are  coded  as  0  and  the  of 
FASA/FAR  1 5  rewrite  years  are  coded  as  1 . 
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Use  of  Multiple  Regression  Techniques 


Chapters  I  and  II  discuss  the  variety  of  factors  that  influence  the  likelihood  of  a 
contract  protest;  therefore,  it  is  unlikely  that  a  simple  straight-line  model  provides  a 
realistic  probability  model.  Instead  of  using  one  predictor  variable  to  estimate  values  on 
the  dependent  variable,  multiple  regression  allows  the  use  of  several  predictor  variables 
(Kachigan,  1986:  259).  The  goal  of  this  analysis  is  two-fold:  (1)  prediction  the  AF  GAO- 
Protests  Processed  based  on  the  independent  variables,  and  (2)  an  assessment  of  the 
variance  to  which  each  independent  variable  accounts  for  in  the  dependent  variable 
(Kachigan,  1986:  265).  As  the  study  proceeds  to  model  formulation  (as  specified  in 
Figure  2),  the  application  of  multiple  regression  techniques  yield  statistical  results. 

The  first  step  is  to  hypothesize  a  model  relating  AF  GAO-Protests  Processed  to 
the  independent  variables  (AF  Contract  Actions,  AF  Contract  Dollars,  AF-GAO  Sustain 
Rate,  U.S.  Unemployment  Rate,  Acquisition  Reform).  This  step  of  the  data  analysis 
consists  of  formulating  a  general  linear  model  for  multiple  regression  as  follows: 
y  =  po  +  Pixi  +  p2X2  +  P3X3  +  P4X4  e 

Next,  the  study  estimates  the  model  coefficients  based  on  the  sample  of  AF  GAO- 
Protests  Processed  (1984-1998).  The  study  uses  the  JMP  Statistical  Software  package  to 
estimate  the  model  coefficients  based  on  the  independent  variables  (xi,  x2,...X5).  This 
procedure  provides  the  estimated  betas  for  the  least  squares  model  and  the  F  statistic/P- 
Value  to  analyze  overall  model  usefulness. 

An  integral  part  of  the  multiple  regression  testing  in  Figure  2  is  the  probability 
distribution  of  the  random  error  component  (e).  Multiple  regression  procedures  specify 
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that  e  is  normally  distributed  with  a  mean  of  zero  and  a  constant  variance  a2  (this  study 
represents  this  assumption  with  the  following  symbolism  [e  ~  N  (0,  a2)]).  The  data 
analysis  chapter  formulates  an  estimate  of  the  variance  (s2)  and  the  standard  deviation 
using  statistical  methods.  Consistent  with  the  normality  assumption,  the  study 
demonstrates  that  the  majority  of  y  values  (AF  GAO-Protests  Processed)  fall  within  2s  of 
the  least  squares  predicted  values. 

The  next  step  of  the  data  analysis  consists  of  determining  the  usefulness  of  the 
model  in  predicting  protests.  Useful  does  not  necessarily  equate  to  the  best  model; 
another  model  may  prove  more  useful  in  terms  of  providing  estimates  that  are  more 
reliable.  This  study  uses  the  testing  heuristic  described  in  the  next  section. 

Testing  Heuristic  Used.  The  study  uses  the  V-Heuristic  to  organize  all  hypothesis 
testing,  state  necessary  assumptions,  and  make  knowledge/value  claims  (Reynolds, 

1998).  The  test  is  broken  down  as  follows: 

1 .  State  assumptions.  In  this  study,  the  following  shall  apply  to  model  evaluation: 

e  ~  N(0,  a2) 
a  =  .05 

2.  State  null  (Ho)  and  alternate  (Ha)  hypothesis. 

3.  Define  the  test  statistic.  The  F  statistic  and  P- Value  shall  be  used. 

4.  State  decision  rule.  The  criteria  as  related  to  the  null/altemate  hypothesis. 

5.  Input  data  and  run  the  statistical  software  program  (JMP). 

6.  Make  knowledge  claim  based  on  the  test  statistic. 

7.  Make  value  claim  regarding  model  usefulness  based  on  evidence. 

8.  Conduct  aptness  test  of  the  assumptions  (Normality  of  Residuals). 
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Conduct  t-tests  on  the  Individual  Betas.  Once  the  usefulness  of  the  multiple 
regression  model  is  determined,  the  study  measures  the  contribution  of  each  independent 
variable  on  model  usefulness.  Since  the  major  focus  of  the  study  is  acquisition  reform, 
this  qualitative  variable  is  subject  to  the  first  t-test.  Following  this  test,  the  study 
measures  the  contribution  of  the  four  quantitative  independent  variables  to  model 
usefulness.  The  section  culminates  with  a  breakdown  of  the  results  of  the  hypothesis 
testing. 

The  Selected  Model  and  Model  Aptness.  From  statistical  analysis,  the  researcher 
selects  the  model  for  estimation  and  prediction.  Certain  independent  variables  contribute 
to  the  usefulness  of  the  model;  however,  statistically  insignificant  independent  variables 
are  removed  from  the  model.  The  squared  multiple  correlation  R  relays  the  portion  of 
the  variance  of  the  dependent  variable  accounted  for  by  all  the  predictor  variables 
combined  (Kachigan,  1986:  261). 

From  JMP  Statistical  output,  the  study  derives  the  least  squares  model  to  predict 
values  for  comparison  against  the  actual  values  of  the  dependent  variable  .  These  values 
provide  the  basis  for  the  residual  analysis  and  test  of  model  assumptions  (also  described 
as  the  test  of  model  aptness).  The  study  tests  the  model  assumptions  with  Wilk-Shapiro 
test  of  normality  and  a  plot  of  the  residuals. 

Following  the  model  aptness  test,  a  breakdown  of  statistical  results  provides  an 
objective  summary  of  the  multiple  regression  analysis.  The  data  analysis  supports  the 
statistical  claims  for  use  in  the  conclusions/recommendations  of  this  study. 
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Summary  of  Research  Design 

The  methodology  in  Chapter  III  covers  all  research  objectives  with  the  proposed 
research  design  and  data  analysis.  The  research  design  attempts  to  identify  trends  in  AF 
contract  protests,  the  hypothesis  testing  for  the  multiple  regression  model,  and  the 
foundation  for  statistical  knowledge  claims.  Chapter  IV  implements  the  study  procedure 
and  implementation  model  described  in  this  chapter. 
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IV.  Data  Analysis 


Introduction 

Chapter  IV  presents  the  number  of  AF  GAO-Protests  Processed  from  1984-1998 
collected  from  SAF/AQCX.  The  first  section  identifies  overall  trends  or  patterns  in  the 
number  of  protests  filed  based  on  graphical  representation  and  descriptive  statistics. 
Next,  a  series  of  scattergrams  between  the  dependent  variable  and  the  quantitative 
independent  variables  indicate  the  possibility  of  a  relationship.  The  study  moves  to 
multiple  regression  analysis  by  presenting  the  necessary  assumptions  for  the  use  of  a 
model  and  hypothesis  testing.  Furthermore,  specific  t-tests  of  the  qualitative  variable 
(acquisition  reform)  and  the  four  quantitative  variables  demonstrate  their  contribution  to 
the  model.  Next,  the  study  removes  the  statistically  insignificant  independent  variables 
and  conducts  regression  analysis  with  the  selected  model.  The  chapter  concludes  with  a 
breakdown  of  statistical  results  on  which  to  base  conclusions  and  recommendations. 

Analysis  of  the  AF-GAO  Protest  Numbers  11984  to  1998) 

A  linear  graph  of  AF-GAO  protests  in  Figure  3  will  be  used  to  make  observations 
about  the  general  trends  from  1984-1998.  From  1984  to  1992,  there  appears  to  be  stable 
period  (albeit  higher  level)  of  contractor  protests  with  little  variability.  In  1993,  the 
number  of  protests  jumps  to  nearly  500  for  the  year.  Following  1993,  protests  appear  to 
be  following  a  sharp  downward  trend  and  ultimately  reach  the  lowest  level  of  202  in 


1998. 
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Figure  3:  AF  GAO-Protests  Processed:  1984  to  1998 

Table  1  provides  the  descriptive  statistics  for  a  deeper  analysis  of  protest 
frequency  between  the  specified  years.  The  researcher  performs  graphical  and  numerical 
analysis  to  discern  fluctuations  about  the  mean  relative  to  the  standard  deviation.  The 
mean  number  of  AF-GAO  Protests  between  1984  and  1998  is  approximately  370  with  a 
standard  deviation  of  88.  In  addition,  the  minimum  value  of  the  range  occurs  in  1998 
(also  two  standard  deviations  outside  the  mean). 

The  Wilk-Shapiro  test  of  normality  results  in  a  score  of  .90  (for  the  dependent 
variable);  this  indicates  that  the  number  of  AF  GAO-Protests  Processed  follows  an 
approximately  normal  distribution.  With  this  information,  the  researcher  notes  that 
approximately  68%  of  the  observations  should  fall  within  one  standard  deviation  (lcr)  of 
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the  mean.  Further  analysis  reveals  that  the  following  years  (mainly  affected  by 
FASA/FAR  15  rewrite)  fall  outside  of  la:  1993,  1996, 1997,  and  1998. 


Table  1.  Protests  (Descriptive  Statistics)  1984  - 1998 


Mean 

369.6 

Standard  Error 

22.6722 

Median 

381 

Mode 

421 

Standard  Deviation 

87.8089 

Sample  Variance 

7710.4 

Kurtosis 

Skewness 

-0.8506 

Range 

290 

Minimum 

203 

Maximum 

493 

Sum 

5544 

Count 

15 

The  descriptive  statistics  provide  measures  of  central  tendency  and  dispersion  of 
the  dependent  variable.  Such  measures  allow  the  researcher  to  learn  the  tendencies  of  the 
distribution  and  highpoints/lowpoints  of  significance.  From  the  descriptive  statistics,  this 
study  concludes  that: 

•  The  AF  GAO-Protests  Processed  follow  an  approximately  normal  distribution. 

•  Years  outside  of  1  a  of  the  mean  warrant  further  investigation. 

•  The  years  associated  only  with  CICA  produced  a  relatively  stable  distribution. 

•  AF  GAO-Protests  Processed  have  been  following  a  downward  trend  since  ’93. 

•  1993  marked  the  peak  of  protests  filed  and  1998  marked  the  low-point. 

•  Multiple  factors  may  be  responsible  for  the  variation  between  the  years. 
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The  next  section  presents  the  a  regression  analysis  of  possible  causes  of  variation. 
Summary  of  Data  Collected 

SAF/AQCX  and  the  United  States  Statistical  Abstract  serves  as  the  source  of  data 
for  the  dependent  and  independent  variables.  Table  2  represents  a  consolidation  of  the 
data  associated  with  the  dependent  and  independent  variables.  The  number  of  AF 
Contract  Actions  and  Dollars  in  1984, 1985,  and  1986  were  unavailable  for  analysis; 
however,  all  other  data  was  obtainable.  Table  2  expresses  the  AF  Contract  Dollars  in 
terms  of  ’92  dollars  (Chapter  III  specifies  conversion  method). 


Table  2.  Dependent  and  Independent  Variables 


PROTESTS 

SUSTAINED 

CONTRACT 
ACTIONS 
(in  millions) 

CONTRACT 
DOLLARS 
(in  millions) 

UNEMP 

RATE 

ACQ 

REFORM 

373 

14 

* 

* 

7.40% 

No 

‘  457 

24 

* 

* 

7.10% 

No 

438 

20 

* 

* 

6.90% 

No 

381 

39 

5.106M 

55. 1M 

6.10% 

No 

390 

26 

4.927M 

53.7M 

5.40% 

No 

421 

13 

5.556M 

55.1  M 

5.20% 

No 

380 

19 

5.709M 

54.4M 

5.40% 

No 

421 

21 

4.933M 

59.6M 

6.60% 

No 

440 

16 

3.135M 

40.6  (Base) 

7.50% 

No 

493 

8 

3.008M 

47.7M 

6.90% 

No 

368 

4 

2.777M 

43.2M 

6.10% 

Yes 

331 

7 

2.546M 

38.2M 

5.60% 

Yes 

246 

3 

2.12M 

41 .6M 

5.40% 

Yes 

212 

5 

1.719M 

33.9M 

4.70% 

Yes 

203 

3 

1.2M 

30.7M 

4.50% 

Yes 
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Graphical  Analysis  of  Dependent/Independent  Variables 

Prior  to  applying  the  multiple  regression  techniques,  the  study  constructs  a  series 
of  scattergrams  to  determine  the  relationship  between  AF  GAO-Protests  Processed  (y) 
and  Contract  Awards  (xi),  Contract  Dollars  (X2),  the  GAO  Sustain  Rate  (X3),  and  the 
Unemployment  Rate  (X4).  The  researcher  determines  if  a  positive,  negative,  or  no 
relationship  exists  between  the  dependent  and  independent  variables.  In  addition,  the 
shape  of  the  data  in  the  scattergram  indicates  the  likelihood  of  a  linear  or  quadratic 
relationship. 

Single  Quantitative  Independent  Variable  #1 :  AF  Contract  Actions.  A 
scattergram  below  depicts  the  possible  relationship  between  protests  and  contract  actions. 
A  visual  inspection  of  Figure  4  reveals  a  possible  relationship  between  the  variables. 
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The  scatterplot  suggests  there  is  a  general  tendency  for  the  number  of  protests  to  increase 
as  the  number  of  AF  Contract  Actions  increase;  however,  the  relationship  may  not  be 
linear.  When  the  number  of  actions  approach  4  million,  the  number  of  protests 
experience  a  decrease. 

Single  Quantitative  Independent  Variable  #2:  AF  Contract  Dollars.  The 
scattergram  (Figure  5)  presents  the  possible  relationship  between  protests  and  contract 
dollars  for  analysis. 


Protests  vs.  AF  Contract  Dollars 


AF  Dollars  ('92  constant) 


Figure  5:  AF  GAO-Protests  Processed  vs.  AF  Contract  Dollars  Scattergram 


A  visual  inspection  of  Figure  5  reveals  a  possible  relationship  between  AF-GAO  Protests 
vs.  AF  Contract  Dollars  Scattergram  the  variables.  The  scatterplot  suggests  there  is  a 
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general  tendency  for  the  number  of  protests  to  increase  as  the  number  of  AF  Contract 
Dollars  increases;  however  the  relationship  may  not  be  linear.  The  number  of  protests 
increases  in  a  linear  manner  up  to  $50  billion  dollars;  however,  they  experience  a  slight 
decline  after  this  point. 

Single  Quantitative  Independent  Variable  #3:  AF-GAO  Sustain  Rate.  The  study 
presents  a  scattergram  to  analyze  the  possible  relationship  between  protests  and  AF-GAO 
sustain  rate.  A  visual  inspection  of  Figure  6  does  not  reveal  a  clear  relationship  between 
the  variables. 


Protests  vs.  Sustain  Rate 


GAO  Sustain  Rate 


Figure  6:  AF  GAO-Protests  Processed  vs.  AF-GAO  Sustain  Rate  Scattergram 

The  scatterplot  suggests  there  may  be  a  general  tendency  for  the  number  of  protests  to 
increase  as  the  number  of  GAO  Sustain  rate  increases;  however,  the  relationship  appears 
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to  be  weak.  While  a  linear  relationship  is  highly  questionable,  there  may  be  evidence  of  a 
quadratic  fit  in  the  data. 


Single  Quantitative  Independent  Variable  #4:  Unemployment  Rate.  A 
scattergram  displays  the  possible  relationship  between  protests  and  the  unemployment 
rate  for  analysis.  A  visual  inspection  of  Figure  7  reveals  a  possible  relationship  between 


Protests  vs  Unemployment  Rate 


4.0%  5.0%  6.0%  7.0%  8.0% 

Unemployment  Rate 


Figure  7:  AF  GAO-Protests  Processed  vs.  Unemployment  Rate  Scattergram 

the  variables.  The  scatterplot  suggests  there  is  a  general  tendency  for  the  number  of 
protests  to  increase  as  the  unemployment  rate  increases  and  the  relationship  appears  to  be 
linear. 
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Using  Multiple  Regression  to  Estimate  AF  GAO-Protests  Processed 


Table  2  specifies  the  dependent  variable  and  independent  variables  used  to 
develop  the  multiple  regression  model.  The  data  obtained  from  SAF/AQCX  and  the 
United  States  Statistical  Abstract  provides  the  necessary  information  to  estimate  the 
model  coefficients  (the  p  parameters).  The  following  subsections  breakdown  each  step 
of  the  model  formulation  and  subsequent  testing  in  the  study. 


Step  1 :  Hypothesize  The  Model  Relating  Protests  to  the  Independent  Variables. 

The  first  step  is  to  hypothesize  a  model  relating  AF  GAO-Protests  Processed  to  the 

independent  variables  listed  in  Table  2.  Chapter  II  presents  a  justification  for  the 

inclusion  of  the  independent  variables  as  factors  affecting  the  decision  to  file  a  protest. 

The  study  proposes  the  following  model: 

y  =  p0  +  pixi  +  p2x2  +  p3x3  +  P4X4  +  P5X5  +  e 

y  =  AF  GAO-Protests  Processed 
Xi  =  AF  Contract  Awards 
X2=  AF  Contract  Dollars 
X3  =  AF-GAO  sustain  rate 

X4  =  U.S.  unemployment  rate  (General  Economic  Condition) 

X5  =  0  if  CICA  prior  to  FASA/FAR  15  rewrite;  1  if  CICA  after 
FASA/FAR  1 5  rewrite 
g  =  random  error  component 
Po  =  y-intercept  of  the  line 


The  variable  X5  is  a  dummy  variable  used  to  describe  an  independent  variable  (acquisition 
reform)  not  measured  on  a  numerical  scale;  it  is  qualitative  (categorical)  in  nature. 

Step  2:  Estimating  the  Model  Coefficients.  In  order  to  estimate  the  model 
coefficients,  a  sample  of  AF-GAO  protests  (1984-1998)  and  the  respective  independent 


variables  (xi,  x2,...x5)  are  inputted  into  the  JMP  statistical  software.  Table  3  lists  the 
output  for  the  multiple  regression. 


Table  3.  JMP  Printout  for  Multiple  Regression  (AF  GAO-Protests  Processed) 


Response:  Protests 

Summary  of  Fit 

RSquare 

0.952286 

RSquare  Adj 

0.912525 

Root  Mean  Square  Error 

27.28684 

Mean  of  Response 

357.1667 

Observations  (or  Sum  Wgts) 

12 

Parameter  Estimates 

Term 

Estimate  Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

20.272254  104.3099 

0.19 

0.8523 

Actions 

69.518997  24.77486 

2.81 

0.0309 

Dollars 

-1.146441  2.869498 

-0.40 

0.7033 

Sust  Rate 

-23.81523  6.47227 

-3.68 

0.0103 

Unemp 

42.747369  14.31356 

2.99 

0.0244 

Acq  Reform[0-1]  36.40437  18.64652 

1.95 

0.0987 

Anaivsis  of  Variance 

Source  DF 

Sum  of  Squares  Mean  Square  F  Ratio 

Prob>F 

Model  5 

89162.237  17832.4  23.9499 

.0007 

Error  6 

4467.430  744.6 

C  Total  1 1 

93629.667 

Table  3  highlights  the  estimated  betas,  F  statistic,  and  P-Value  for  ease  of  reference  in  the 
subsequent  analysis.  The  least  squares  model  is: 


y-hat  =  20.27  +  69.52x!  -  1.15x2  -  23.82x3  -  42.75x4  -  36.40xs  +  e 
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Step  3:  Probability  Distribution  of  the  Random  Error  Component  (e).  Next, 


the  analysis  specifies  the  probability  distribution  of  the  random  error  component. 
Multiple  regression  procedures  require  that  e  is  normally  distributed  with  a  mean  of  zero 
and  a  constant  variance  <7  [e  ~  N  (0,  a2)]).  In  addition,  the  errors  are  assumed 
independent.  The  estimate  of  the  variance,  shown  in  Table  3  as  Mean  Square  (Residual), 
is: 


s2  =  SSE  =  4467.4430  =  744.57 

N-(k+l)  12 -(6) 

The  estimate  of  the  standard  deviation  on  the  JMP  printout  as  standard  error  is  s  =  Vs2 
which  is  27.286.  This  means  that  the  majority  of  y  values  (AF  GAO-Protests  Processed) 
fall  within  2s  =  2(27.286)  =  54.57  of  the  least  squares  predicted  values. 

Step  4:  Evaluate  Usefulness  of  the  Model.  The  next  step  of  the  data  analysis 
consists  of  determining  the  usefulness  of  the  model  in  predicting  protests.  In  summary, 
the  model  must  pass  the  overall  F-test  to  merit  further  consideration  in  predicting  the 
number  of  protests  (McLave  and  Benson,  1998:  579).  Using  the  V-Heuristic  (Reynolds, 
1998),  the  elements  of  the  overall  test  of  the  model  follows. 

a)  Assumptions:  The  standard  regression  assumptions  about  the  random  error 
component 

apply. 

e  ~  N  (0,  <t2) 

The  alpha  level  for  the  test  is:  a  =  .05 

b)  Ho  :  Pi  =  P2  =•  •  .=Ps  =  0  Ha  :  At  least  one  of  the  model  coefficients  is 

nonzero. 

Y  =  p0  +  s  Y  =  p0  +  Pi(xi)  +  p2(x2)  +  p3(x3)  +  p4(x4)  +  p5(x5)  + 

s  Reduced  Model  Full  Model 
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c)  Test  statistic:  F* 


=  Mean  Square  (Model)  =  23.9499  (See  Table  3) 

MSE 

F.05  =  V i  Parameter  =  2 

V2  Parameter  =  n  -  (k  +1)  =  12-3  =  9 
F.05  ~  4.26  (from  Table  IX  of  Appendix  B  in  McLave-Benson) 

d)  Decision  Rule:  If  F*  is  greater  than  F.05  the  null  hypothesis  shall  be  rejected;  else  fail 
to  reject  the  null  hypothesis.  If  the  P-Value  is  greater  than  a  =  .05  the  null  hypothesis 
shall  be  rejected;  else  fail  to  reject  the  null  hypothesis. 

e)  Knowledge  Claim:  F*  =  23.9499  >  F,05  *  4.26;  therefore  F*  does  fall  in  the  rejection 
region.  As  the  testing  indicates  that  one  of  the  betas  differ  from  zero;  the  researcher 
rejects  the  null  hypothesis  at  an  alpha  level  of  .05. 

f)  Value  Claim:  There  is  sufficient  evidence  to  indicate  that  the  overall  model  is  useful 
in  predicting  the  number  of  AF  GAO-Protests  Processed;  however,  further  testing  is 
required  to  determine  the  contributions  by  each  independent  variable. 

Step  5:  Testing  the  Contributions  of  the  Independent  Variables  to  Model 
Usefulness.  If  the  acquisition  reforms  associated  with  FAS  A/FAR  15  rewrite 
influenced  the  AF  GAO-Protests  Processed,  the  mean  number  of  protests  differ  for  the 
respective  periods  (represented  by  the  qualitative  variable).  Does  the  data  provide 
evidence  that  the  number  of  protests  change  in  response  to  acquisition  reforms 
symbolized  by  FAS  A  and  the  FAR  15  rewrite  (efficiency  changes)?  This  test  holds  the 
quantitative  independent  variables  constant  and  the  following  hypothesis  testing  applies: 

a)  Hypotheses  for  testing:  H0 :  P5  =  0  HA :  Ps  >  0 

b)  Test  statistic:  t*  =  36.40437  =  1.95  (See  Table  3) 

18.64652 

t  os  (with  5  degrees  of  freedom)  =  2.015 

(From  inside  cover  table  in  McLave-Benson) 


62 


c)  Decision  Rule:  If  t*  is  greater  than  1 05  the  null  hypothesis  shall  be  rejected;  else  fail  to 
reject  the  null  hypothesis.  If  the  P-Value  is  greater  than  a  =  .05  the  null  hypothesis  shall 
be  rejected;  else  fail  to  reject  the  null  hypothesis. 

c)  Knowledge  Claim:  t*  =  1 .95  <  1 05 »  2.01 5;  therefore  t*  does  not  fall  in  the  rejection 
region.  Therefore,  the  null  hypothesis  is  not  rejected  at  an  alpha  level  of  .05.  In 
addition,  the  observed  significance  level  (.0987)  exceeds  alpha  (.05);  evidence  does 
not  exist  to  reject  the  null  hypothesis. 

d)  Value  Claim:  There  is  not  sufficient  evidence  to  indicate  that  the  acquisition  reform 
is  useful  in  predicting  the  number  of  AF  GAO-Protests  Processed  in  the  multiple 
regression  model. 

Measuring  the  Contribution  of  the  Quantitative  Independent  Variables 

This  section  presents  a  breakdown  of  the  t-ratios  and  significance  level  of  the 
quantitative  variables  for  evaluation.  As  with  the  acquisition  reform  hypothesis  testing, 
the  values  are  compared  with  the  t.05  =  2.015  and  a  =  .05.  Table  4  presents  a  breakdown 
of  the  results  of  the  hypothesis  testing: 
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Table  4.  Utility  of  Quantitative  Variables  in  Multiple  Regression  Model 


Variable 

t-ratio 

+■» 

A 

-Q 

2 

o. 

Useful  to  Multiple 
Regression  Model? 

Actions 

2.81 

0.0309 

YES 

Dollars 

-0.4 

0.7033 

NO 

Sust  Rate 

-3.68 

0.0103 

YES 

hh 

2.99 

0.0244 

YES 

From  Table  4,  the  study  concludes  that  AF  Contract  Action,  the  GAO  Sustain 
Rate,  and  the  Unemployment  Rate  (general  economic  conditions)  specifically  contribute 
to  the  usefulness  of  the  model.  On  the  other  hand,  AF  Contract  Dollars  are  not 
statistically  significant  in  the  multiple  regression  model;  therefore,  the  variable  is 
removed  for  the  next  step  of  the  data  analysis. 

Step  6:  Using  the  Model  for  Estimation/Prediction — The  Selected  Model.  The 

statistical  analysis  culminates  with  the  test  of  the  model  for  estimation  and/or  prediction. 

AF  Contract  Dollars  and  Acquisition  Reform  do  not  contribute  to  the  usefulness  of  the 

model;  therefore,  they  have  been  removed.  For  this  step,  the  researcher  selects  the 

following  multiple  regression  model  for  estimation/prediction: 

y  =  Po  +  Pixi  +  P2X2  +  P3X3  +  e 

y  =  AF  GAO-Protests  Processed 
xj  =  AF  Contract  Awards 
X2  =  AF-GAO  sustain  rate 

x3  =  U.S.  unemployment  rate  (General  Economic  Condition) 
e  =  random  error  component 
Po  =  y-intercept  of  the  line 
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Table  5  depicts  the  results  of  the  regression  analysis. 


Table  5.  JMP  Printout  for  Selected  Model  (AF  GAO-Protests  Processed) 


Response:  Protests 

SummarvofFit 

RSquare 

0.91921 

RSquare  Adj 

0.888913 

Root  Mean  Square  Error 

30.74976 

Mean  of  Response 

357.1667 

Observations  (or  Sum  Wgts) 

12 

Parameter  Estimates 

Term 

Estimate  Std  Error 

t  Ratio 

Prob>|t| 

VIF 

Intercept 

-148.9268  62.97584 

-2.36 

0.0456 

0 

Actions 

74.179263  16.9237 

4.38 

0.0023 

8.45 

Sust  Rate 

-21.00627  7.05774 

-2.98 

0.0177 

7.97 

Unemp 

58.499826  11.71663 

4.99 

0.0011 

1.22 

Analysis  of  Variance 

Source  DF 

Sum  of  Squares 

Mean  Square 

F  Ratio  Prob>F 

Model  3 

86065.286 

28688.4 

30.3405  0.0001 

Error  8 

7564.380 

945.5 

C  Total  11 

93629.667 

When  compared  to  the  full  model  used  in  Table  3,  the  selected  model  displays  a  higher  F- 
Ratio  and  a  lower  significance  level.  This  improvement  in  the  model  (by  removing  the 
acquisition  reform  and  AF  Contract  Dollars)  is  evident  with  this  change  in  the  statistics 
measuring  overall  usefulness.  In  addition,  R2  =  .919  implies  over  91%  of  the  variation  in 
AF  GAO-Protests  Processed  for  the  years  sampled  can  be  accounted  for  by  this  model. 
From  Table  5,  the  study  derives  the  following  least  squares  model: 
y-hat  =  -148  +  (74.2)x!  -  (21)x2  +  (58.5)x3  +  € 

Table  6  contains  the  predicted  values  versus  the  actual  values  associated  with  the  model. 
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Table  6.  AT  GAO-Protests  Processed  (Predicted  vs.  Actual) 


Year 

Actual  Protests 

Predicted  Protests 

1987 

381 

377.4272 

1988 

390 

398.1916 

1989 

421 

398.7001 

1990 

380 

386.6373 

1991 

421 

435.1947 

1992 

440 

445.9111 

1993 

493 

443.8231 

1994 

368 

390.3909 

1995 

331 

323.2094 

1996 

246 

298.6046 

1997 

212 

215.6617 

1998 

203 

172.2483 

Residual  Analysis — Model  Aptness  Testing.  The  study  tests  the  model 
assumptions  with  the  Wilk-Shapiro  test  of  normality  and  a  plot  of  the  residuals.  Table  7 
shows  a  Wilk-Shapiro  score  of  .964;  consistent  with  the  assumptions. 
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Table  7.  Normality  Test  of  Residuals  (Selected  Model) 

Shapiro-Wilk  W  Test - 

W  Prob<W 

0.964099  0.7850 


The  plot  of  the  residuals  in  Figure  8  reveals  scatter  above  and  below  the  plane  in  the 
residual  plot.  There  appears  to  be  a  random  pattern  of  residuals  around  the  horizontal  0 
(mean  of  the  residuals);  therefore,  the  selected  model  stands  to  provide  an  adequate 
source  for  predicting  AF  GAO-Protests  Processed. 
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Summary  of  Statistical  Results 

An  application  of  multiple  regression  techniques  to  determine  whether  Air 
Force  contract  protest  frequency  rises  or  falls  with  changes  in  the  five  independent 
variables  yields  the  following  statistical  results: 

•  There  is  sufficient  evidence  to  indicate  that  AF  Contract  Actions  may  be 
useful  in  predicting  the  number  of  AF-GAO  Protests-Processed 

•  There  is  insufficient  evidence  to  indicate  that  AF  Contract  Dollars  may  be 
useful  in  predicting  the  number  of  AF-GAO  Protests-Processed 

•  There  is  sufficient  evidence  to  indicate  that  GAO  Sustain  Rate  is  useful  in 
predicting  the  number  of  AF-GAO  Protests-Processed 

•  There  is  sufficient  evidence  to  indicate  that  the  Unemployment  Rate 
(General  Economic  Condition)  may  be  useful  in  predicting  AF-GAO 
Protests-Processed 

•  There  is  insufficient  evidence  to  indicate  that  Acquisition  Reform  is  useful 
in  predicting  the  number  of  AF-GAO  Protests-Processed 

In  order  to  make  an  accurate  assessment  of  the  independent  variables'  effect  on 
bid  protests;  this  data  analysis  provides  an  objective  look  at  protest  trends  and  associated 
conditions  to  the  Air  Force.  While  the  results  of  the  research  have  not  established  causal 
relationships,  the  statistical  correlation  provides  Air  Force  decision-makers  a  tool  to 
identify  trends  in  contract  protest  frequency. 
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Summary 


This  chapter  provides  a  data  analysis  of  the  protest  trends  from  1984-1998  and 
regression  analysis  of  the  impact  of  quantitative  and  qualitative  variables.  It  offers  a 
breakdown  of  the  statistical  results  and  a  multiple  regression  model  for  use  in  predicting 
AF  GAO-Protests  Processed.  The  analysis  concludes  with  statistical  insight  on  other 
factors  besides  the  acquisition  reform  initiatives  that  may  influence  the  number  of 
protests  filed. 

How  have  the  acquisition  reforms  affected  the  number  of  AF  GAO-Protests 
Processed?  Have  the  number  of  contract  dollars  and  actions  affected  these  numbers  in 
conjunction  with  the  reforms?  Does  the  GAO  sustain  rate  have  a  statistical  impact  on  the 
numbers  of  protests  filed  in  the  AF?  Do  general  economic  conditions  have  any  impact  on 
the  numbers  of  protests  filed  in  the  AF?  Chapter  IV  presents  statistical  results  concerning 
the  listed  questions;  however,  the  next  chapter  draws  conclusions  and  recommendations 
from  these  results. 
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V.  Conclusions  and  Recommendations 


Overview 

This  chapter  restates  the  research  problem  and  objectives  formulated  in  Chapter  I 
to  ensure  adequate  coverage  in  the  study.  Chapter  V  presents  a  condensed  version  of  the 
statistical  results  on  which  to  draw  relevant  conclusions  for  the  AF  decision-maker  and 
acquisition  professional.  Next,  the  researcher  proposes  a  list  of  recommended  areas  of 
study  based  on  the  findings  of  this  thesis.  The  study  concludes  with  a  brief  discussion  of 
the  limitations  of  the  study  and  a  final  summary. 

Revisiting  the  Problem 

The  previous  chapter  uses  statistical  analysis  to  meet  the  research  objectives; 
however,  a  reiteration  of  the  research  problem  puts  the  results  into  appropriate  context. 
This  study  investigates  the  influence  of  AF  Contract  Dollars,  AF  Contract  Actions,  AF- 
GAO  sustain  rate,  Acquisition  Reform  initiatives,  and  general  economic  conditions  on 
the  number  of  AF  GAO-Protests  Processed.  A  quantitative  study  of  these  external  factors 
provides  AF  decision-makers  and  acquisition  professionals  a  tool  to  identify  trends  in 
contract  protest  frequency. 

The  regression  analysis  provides  a  broad  statistical  analysis  of  the  number  of  Air 
Force-GAO  Protests  Processed  as  correlated  to  the  variables.  A  primary  purpose  is  to 
determine  if  acquisition  reform  initiatives  make  a  statistical  impact  on  the  number  of 
protests  filed.  Since  multiple  regression  is  used,  the  researcher  exercises  extreme  caution 
in  interpreting  the  results  of  the  analysis.  The  conclusions  drawn  from  this  study  do  not 
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point  to  the  cause  of  a  protest;  however,  they  direct  attention  to  the  conditions  that  tend  to 
coincide  with  an  increase  in  the  number  of  protests.  Correlation  is  not  causation; 
therefore,  the  results  are  limited  to  the  appropriate  level  of  meaning  for  this  statistical 
analysis. 

Meeting  the  Objectives  of  the  Thesis 

Identify  Trends  in  the  Number  of  Protests  Filed  from  1984  to  1998.  A  linear 
graph  of  AF-GAO  protests  reveals  the  general  trends  from  1984-1998.  From  1984  to 
1 992,  there  was  a  stable  period  (albeit  at  a  higher  level)  of  contractor  protests  with  little 
variability.  From  the  data,  it  appears  that  the  CICA-only  period  resulted  in  an  elevated 
number  of  protests  as  compared  to  the  subsequent  years.  In  1993,  the  number  of  protests 
jumps  to  nearly  500  for  the  year.  FASA  was  implemented  in  1994  and  the  FAR  15 
rewrite  became  effective  in  1997;  however,  the  impact  of  such  reforms  on  the  number  of 
protests  is  unclear. 

Following  1993,  protests  follow  a  sharp  downward  trend  and  ultimately  reach  the 
lowest  point  of  202  in  1998.  While  an  observer  might  point  to  the  introduction  of 
acquisition  reform  as  the  reason  for  the  decline,  an  analysis  of  other  factors  illuminate  the 
complexity  associated  with  the  contractor’s  protest  decision.  It  should  be  noted  that  in 
the  1990s  there  has  been  a  sharp  decline  in  AF  Contract  Actions,  AF  Contract  Dollars, 
and  low  unemployment  (positive  economic  condition).  The  visual  observation  of  the 
protest  trends  warrants  an  analysis  of  these  other  factors’  influence  on  the  number  of 
protests. 
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Identify  the  Protest  Trends  Associated  with  the  AF  Contract  Actions  and  Dollars. 


Chapter  IV  analyzes  a  possible  relationship  between  AF  Contract  Actions  and  the 
AF  GAO-Protests  Processed.  The  scatterplot  suggests  there  is  a  general  tendency  for  the 
number  of  protests  to  increase  as  the  number  of  AF  Contract  Actions  increases;  however, 
the  relationship  may  not  be  linear.  When  the  number  of  actions  approach  4  million,  the 
number  of  protests  experiences  a  decrease.  The  decrease  in  protests  appears  to  flatten  out 
to  around  400  protests  when  AF  Contract  Actions  are  between  5  and  6  million.  Before 
discussing  the  significance  of  these  results,  the  study  analyzes  the  number  of  actions  as 
compared  to  the  dollars  awarded  over  the  period. 

With  regard  to  AF  Contract  dollars,  the  scatterplot  suggests  there  is  a  general 
tendency  for  the  number  of  protests  to  increase  as  the  number  of  AF  Contract  Dollars 
increases;  however  the  relationship  may  not  be  linear.  The  number  of  protests  increases 
in  a  linear  manner  up  to  50  billion  dollars;  however,  they  experience  a  slight  decline  after 
this  point.  When  compared  to  the  number  of  contract  actions,  the  relationship  to  contract 
dollars  bears  a  closer  resemblance  to  a  linear  model. 

The  data  analysis  reveals  that  increases  in  the  number  of  AF  Contract  Actions  and 
Dollars  generally  lead  to  an  increase  in  the  number  of  protests.  Chapter  II  discusses  the 
possibility  of  leaner  budgets  leading  to  an  increased  number  of  protests.  Increases  in  the 
number  of  contract  actions  (as  enhancing  the  exposure  to  a  protest)  might  explain  an 
increase  in  the  numbers.  The  data  support  the  assertions  related  to  the  contract  actions; 
however,  the  findings  about  the  dollars  run  contrary  to  the  expected  results.  Instead  of 
thicker  competition  in  the  lean  times  leading  to  an  increased  number  of  protests,  the  AF 
GAO-Protests  Processed  decline.  Contractors  may  perceive  a  larger  pot  of  AF  Dollars 
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available  for  award;  therefore,  they  are  more  likely  to  file  a  protest.  Subsequent  sections 
expand  on  the  statistical  significance  of  the  relationship  between  AF  Contract 
Actions/Dollars  and  AF-GAO  Protest  Processed. 

Identify  the  Protest  Trends  Associated  with  the  AF-GAO  Sustain  Rate.  Chapter 
IV  analysis  reveals  a  possible  relationship  between  the  AF-GAO  sustain  rate  and  the  AF 
GAO-Protests  Processed.  The  scatterplot  suggests  a  general  tendency  for  the  number  of 
protests  to  increase  as  the  GAO  Sustain  rate  increases;  therefore,  the  findings  lend 
credence  to  assertions  in  this  study.  In  Chapter  II,  the  Lieberman  study  brings  the 
significance  of  the  protest  forum  to  light  in  the  contractor’s  decision  to  file  a  protest.  The 
data  support  the  conclusion  that  the  number  of  AF  GAO-Protests  Processed  declines  as 
contractors  encounter  less  success  at  the  GAO. 

A  number  of  lawmakers  and  acquisition  professionals  quickly  point  to  contractor 
consolidation,  fewer  contracts,  and  changes  brought  about  by  acquisition  reform  leading 
to  the  reduction  in  protests.  The  AF-GAO  sustain  rate  is  often  overlooked  as  a 
contributing  factor  to  the  reduction  in  protests.  Even  as  protests  have  declined  in  the  late 
1990s,  the  AF-GAO  sustain  rate  has  reduced  to  below  2%.  The  statistical  analysis 
presents  support  for  the  influence  of  the  AF-GAO  sustain  rate  on  the  number  of  AF 
GAO-Protests  Processed. 

Identify  the  Protest  Trends  Associated  with  General  Economic  Conditions.  The 
data  analysis  shows  the  possible  relationship  between  the  General  Economic  Conditions 
(represented  by  the  unemployment  rate)  and  the  number  of  AF  GAO-Protests  Processed. 
The  scatterplot  suggests  there  is  a  general  tendency  for  the  number  of  protests  to  increase 
as  the  unemployment  rate  increases  and  the  relationship  appears  to  be  linear.  The 
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literature  review  discusses  the  issue  of  whether  an  increase  in  protests  is  associated  with 
perceived  economic  bad  times.  With  the  unemployment  rate  chosen  to  represent 
economic  condition,  the  number  of  protests  tends  to  decline  with  the  unemployment  rate. 
The  researcher  carefully  tempers  conclusions  in  this  area  because  many  indices  exist  to 
measure  the  health  of  the  economy. 

In  periods  of  increased  unemployment,  the  number  of  protests  tends  to  increase. 
Contractors  may  be  experiencing  a  decline  in  commercial  business  (negative  influences 
in  the  economy)  and  be  more  likely  to  challenge  an  AF  contract  award  in  perceived  bad 
times.  Furthermore,  the  periods  of  lower  unemployment  are  associated  with  a  decline  in 
the  AF  GAO-Protests  Processed.  The  perceived  good  times  may  equate  to  improved 
commercial  business  for  contractors;  therefore,  a  failure  to  receive  an  AF  contract  award 
would  be  less  pronounced.  In  any  event,  the  statistical  results  concerning  the 
unemployment  rate  in  the  study  present  additional  support  to  the  assertion. 

Identify  the  Protest  Trends  Associated  with  Acquisition  Reform.  Chapter  II 
discusses  the  dramatic  changes  brought  about  by  CICA  to  the  bid  protest  mechanism. 
With  substantial  new  grounds  under  which  to  file  a  protest,  the  CICA  period  appears  to 
follow  a  consistent  (albeit  higher)  level  of  protest  frequency.  Chapter  II  explains  the 
push  for  equity  and  increased  power  of  the  contractor  in  the  protest  process;  therefore,  the 
researcher  expects  a  higher  level  of  protest  between  1984-1994.  The  data  supports  that 
an  increased  level  of  AF-GAO  Protest  Processed  is  associated  with  the  CICA-only 
period. 

FASA  and  the  FAR  15  rewrite  have  been  classified  as  efficiency  changes  by  this 
study  based  on  the  findings  of  the  literature  review.  In  addition,  the  reforms  of  the  1990s 
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encourage  an  increased  level  of  judgement  by  the  contracting  officer  and  enhanced 
communication  between  the  parties.  AF  GAO-Protests  Processed  reaches  a  highpoint  of 
493  in  1993;  however,  they  decline  to  368  in  the  following  year.  Following  1994  (the 
year  FASA  was  implemented)  protests  continue  to  follow  a  downward  trend. 
Furthermore,  the  FAR  15  rewrite  (implemented  in  1997)  is  associated  with  a  continued 
decline  in  the  number  of  AF  GAO-Protests  Processed.  The  researcher  bases  these 
observations  on  the  graphical  representations  of  the  trends;  however,  a  statistical  analysis 
probes  deeper  into  the  relationship. 

The  Main  Objectives.  The  remaining  objectives  relate  to  the  statistical  analysis 
used  by  this  study  and  form  the  basis  for  conclusions  to  be  drawn  in  relation  to  the 
hypothesis.  These  objectives  are  listed  as: 

•  Determine  if  a  significant  statistical  relationship  exists  between  acquisition 
reform  and  the  number  of  protests  filed  in  the  AF. 

•  Determine  if  a  significant  statistical  relationship  exists  between  the  contract 
actions,  dollars  awarded,  GAO  Sustain  Rate  and  General  Economic  Condition 
with  the  amount  AF  GAO-Protests  Processed. 

•  Test  and  evaluate  the  factors  using  multiple  regression  analysis. 

Chapter  IV  states  the  necessary  assumptions  for  the  use  of  regression  techniques  and  tests 
the  relationships  for  statistical  significance.  The  next  subsections  list  the  propositions 
and  hypotheses  from  Chapter  I  related  to  the  statistical  findings.  From  these  findings,  the 
study  presents  a  discussion  of  the  statistical  results  and  draws  conclusions  for  the  AF 
decision-maker. 
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Discussion  of  Results 


Table  8  provides  a  breakdown  of  the  results  from  Chapter  IV  for  review  and 
convenience  to  the  reader.  The  independent  variables  are  listed  in  the  left  column  with 
the  corresponding  statistical  significance  to  AF  GAO-Protests  Processed. 

Table  8:  Condensed  Results  from  Chapter  IV 


Term 

t-ratio 

Prob>t 

Useful  to  estimation 
and  prediction? 

Actions 

2.81 

0.0309 

YES 

Dollars 

-0.4 

0.7033 

NO 

Sust  Rate 

-3.68 

0.0103 

YES 

Unemployment  Rate 

2.99 

0.0244 

YES 

Acq  Reform  (0-1) 

1.95 

0.0987 

NO 

A  presentation  of  each  hypothesis  and  conclusions  follow  in  the  following  sections. 

Hypothesis  #L  An  increase  in  the  number  of  AF  contract  actions  will 
decrease  the  number  AF  GAO-Protests  Processed  from  1985-1998.  The  regression 
analysis  indicates  the  statistical  significance  of  the  number  of  AF  Contract  Actions  in 
relation  to  the  number  of  AF  GAO-Protests  Processed;  therefore,  the  data  support  the 
existence  of  a  relationship.  Table  8  shows  the  relationship  based  on  multiple  regression 
techniques;  however,  the  nature  of  the  relationship  runs  contrary  to  Hypothesis  #1 .  The 
analysis  shows  a  tendency  for  the  number  of  protests  to  increase  with  the  number  of 
contract  actions  up  to  approximately  3  million;  however,  they  begin  to  decline  beyond 
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4.9  million.  For  this  reason,  the  study  concludes  that  AF-GAO  protests  will  follow  an 
upward  trend  with  the  increase  in  actions;  however,  they  stabilize  at  a  certain  point. 
These  findings  lead  to  a  rejection  of  hypothesis  #1;  an  increase  in  contract  actions  does 
not  necessarily  lead  to  a  reduction  in  protests. 

The  decrease  in  the  number  of  AF  GAO-Protests  Processed  as  associated  with 
fewer  actions  may  be  due  to  less  exposure  to  contractors.  Quite  simply,  contractors  may 
have  fewer  opportunities  to  file  a  protest  and/or  AF  Contracting  Personnel  have  fewer 
opportunities  to  incite  a  protest.  This  relatively  simple  explanation  runs  contrary  to  the 
expected  competitive  behavior  to  “win  the  contract;”  therefore,  further  explanation 
should  be  sought. 

The  decline  in  protests  for  the  period  examined  (1984-1998)  has  been  associated 
with  a  high  degree  of  consolidation  in  the  U.S.  defense  industry.  Since  1982,  the  number 
of  major  systems  defense  contractors  has  been  reduced  from  51  to  5  (Boeing,  Raytheon, 
Litton,  Lockheed  Martin,  and  Northrup  Grumman).  With  the  reduced  number  of 
contractors  and  fewer  AF-Contract  Actions,  there  may  have  been  reduced  opportunities 
to  protest.  While  this  may  explain  the  reduction  in  protests  as  AF  Contract  Actions 
decrease,  the  large  numbers  of  non-systems  related  actions  beg  explanation. 

For  the  above-mentioned  reasons,  extreme  caution  should  be  used  when  making 
generalizations  beyond  the  period  examined.  The  data  and  statistical  tests  indicate  that 
contract  actions  may  contribute  to  the  increased  number  of  protests;  however,  this  single 
factor  in  a  unique  period  should  be  viewed  in  the  context  of  the  observations. 
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Hypothesis  #2.  An  increase  in  the  AF  contract  dollars  will  lead  to  a  decrease 
in  AF  GAO-Protests  Processed.  The  regression  analysis  indicates  that  the  number  of 
AF  contract  dollars  is  not  statistically  significant  in  relation  to  the  number  of  AF-GAO 
Protests.  The  number  of  protests  appear  to  follow  an  upward  trend  with  increases  in  AF 
Contract  Dollars.  These  findings  lead  to  rejection  of  hypothesis  #2;  an  increase  of  dollars 
does  not  necessarily  lead  to  a  decrease  in  protests. 

Hypothesis  #2  was  based  on  the  premise  that  increases  in  the  contract  dollars 
available  would  lead  contractors  to  perceive  more  opportunities  for  contracts.  This 
assumes  that  everybody ’s  chance  to  get  a  contract  increases  with  more  dollars,  thereby 
reducing  the  likelihood  of  protest.  The  data  over  the  time  period  covered  appear  to  run 
contrary  to  this  assumption. 

Table  8  shows  the  statistical  insignificance  of  AF  Contract  Dollars  to  AF  GAO- 
Protests  Processed  between  1984-1998.  The  lack  of  a  relationship  may  be  due  to  the 
Department  of  Defense  budget  procedure.  AF  Contracting  personnel  and  federal 
contractors  exhibit  the  least  amount  of  control  in  determining  the  budget;  therefore,  the 
AF  Contract  Dollars  may  be  seen  as  less  of  a  factor.  On  the  other  hand,  the  contracting 
parties  enjoy  a  higher  degree  of  control  with  AF  Contract  Actions  (i.e.  AF  Contracting 
determines  nature  of  the  business  activities  and  the  contractors  compete  for  these  contract 
actions). 

As  mentioned  in  hypothesis  #1,  the  study  must  be  viewed  with  increased  care  due 
to  the  unique  nature  of  the  period  covered.  The  data  and  statistical  tests  indicate  that 
contract  dollars  may  not  contribute  to  the  increased  number  of  protests;  however,  broad 
generalizations  beyond  this  period  should  be  tempered  with  multivariate  analysis. 
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Hypothesis  #3.  An  increase  in  AF-GAO  sustain  rate  can  be  associated  with 
an  increase  in  the  number  AF  GAO-Protests  Processed.  The  regression  analysis 
indicates  that  the  number  of  AF-GAO  sustain  rate  is  statistically  significant  in  relation  to 
the  number  of  AF-GAO  Protests.  The  data  supports  hypothesis  #3  with  a  correlation 
between  the  rise  and  fall  of  the  mentioned  variables.  In  summary,  it  appears  that  protest 
frequency  is  linked  to  the  contractors’  likelihood  of  success  at  the  GAO. 

With  this  hypothesis  accepted,  the  researcher  asks:  do  AF  GAO-Protests 
Processed  decline  because  the  AF-GAO  Sustain  rate  increases?  Table  8  reveals  that  the 
AF-GAO  Sustain  rate  bears  the  strongest  relationship  to  the  AF  GAO-Protests  Processed. 
In  Chapter  II,  Lieberman  predicted  a  reduction  in  the  protest  numbers  with  less  chance  of 
relief  at  the  GAO;  the  data  support  this  assertion. 

Hypothesis  #4.  Periods  of  higher  unemployment  can  be  associated  with 
increases  in  the  number  of  AF  GAO-Protests  Processed.  The  data  analysis  shows  the 
statistical  significance  of  the  U.S.  unemployment  rate  (general  economic  conditions)  in 
relation  to  the  AF  GAO-Protests  Processed  over  the  specified  period.  Table  8  displays 
the  strong  relationship  (secbnd  only  to  the  AF-GAO  Sustain  Rate)  and  Chapter  IV’ s 
graphical  analysis  demonstrates  the  increased  number  of  protests  associated  with  a  higher 
unemployment  rate.  Consequently,  the  data  support  the  relationship  discussed  in 
Hypothesis  #4. 

From  the  data,  the  researcher  concludes  that  a  downturn  in  the  economy  (signified 
by  an  increase  in  the  unemployment  rate)  may  bring  about  an  increase  in  the  number  of 
protests.  The  perceived  bad  times  would  cause  the  contractor  to  battle  for  economic 
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survival  in  the  commercial  and  public  sectors  of  the  economy.  In  addition,  periods  of 
relative  prosperity  and  good  times  in  the  economy  (signified  by  a  decrease  in  the 
unemployment  rate)  will  be  associated  with  fewer  protests. 

These  conclusions  should  be  viewed  with  care.  The  study  used  the 
unemployment  rate  to  gauge  general  economic  condition;  however,  multiple 
macroeconomic  factors  exist  to  measure  economic  performance. 

Hypothesis  #5.  The  introduction  of  FASA  and  FAR  15  rewrite  (efficiency 
changes)  did  not  reduce  the  mean  number  of  AF  GAO-Protests  Processed.  The 
regression  analysis  indicates  that  FASA/FAR  15  rewrite  (efficiency  changes)  is  not 
statistically  significant  in  relation  to  the  number  of  AF-GAO  Protests.  In  the  regression 
analysis,  dummy  variables  represent  periods  of  equity  versus  efficiency  reforms.  The  0-1 
coding  tested  for  a  relationship  to  the  slope  of  the  line  in  the  multiple  regression  model; 
therefore,  the  influence  on  these  periods  was  categorized.  Based  on  the  statistical  tests, 
hypothesis  #6  should  not  be  rejected;  however,  major  limitations  in  the  data  limit 
application  of  the  results. 

During  the  period  associated  with  CICA,  AF-GAO  Protests  follow  a  relatively 
stable  trend.  A  sharp  decrease  occurs  in  the  period  following  1993-1994;  however,  the 
statistical  relationship  to  FASA  or  the  FAR  15  rewrite  is  insignificant.  While  the  data 
should  not  be  used  to  claim  that  acquisition  reform  is  insignificant,  the  researcher 
concludes  that  attributing  the  downturn  in  protests  to  policy  changes  may  be  simplistic. 

A  variety  of  factors  contribute  to  the  decision  to  file  a  protest;  reforming  the 
protest  system  or  rewriting  sections  of  the  FAR  may  not  be  the  strongest  influence.  In 


80 


viewing  the  protest  numbers,  increases  or  decreases  may  be  due  to  external  factors 
beyond  the  control  of  lawmakers  or  acquisition  professionals. 

Cantor  contends  that  lawmakers  may  have  reached  a  point  of  “leaving  well 
enough  alone”  with  the  bid  protest  process  (Cantor,  1997: 155).  Table  8  shows  the 
statistical  insignificance  of  acquisition  reform  relative  to  other  external  factors;  therefore, 
these  findings  may  support  Cantor’s  assertion.  Cantor  also  warns  that  sanctions  against 
frivolous  protests  and  “efficiency  changes”  promise  to  discourage  the  number  of  protests. 
Again,  the  statistical  findings  do  not  support  the  acquisition  reforms  associated  with 
efficiency  as  having  a  chilling  effect  (term  used  by  Cantor)  oh  the  number  of  protests. 

The  researcher  concludes  that  the  reasons  behind  the  decision  to  file  a  protest  are 
complex  and  dynamic;  therefore,  it  is  difficult  to  influence  the  behavior  with  a  series  of 
reforms.  The  study  does  not  pronounce  a  verdict  regarding  the  efficiency  reforms; 
however,  the  variety  of  external  factors  (independent  of  acquisition  reform)  influencing 
the  protest  decision  are  brought  to  light.  With  regard  to  reforming  the  bid  protest  system 
or  initiating  a  plethora  of  reforms,  lawmakers  should  address  the  value  added  by 
additional  legislation. 

Implications  for  the  AF  Decision-Maker/Acquisition  Professional 

The  implications  of  this  study  to  AF  decision-makers  and  acquisition 
professionals  will  be  discussed  on  the  respective  levels.  Both  groups  should  recognize 
that  this  study  did  not  process  numbers  from  a  regression  analysis  to  find  the  root  cause 
of  the  AF  GAO-Protests  Processed.  The  study  provides  a  tool  to  aid  in  understanding  the 
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complexity  of  the  decision  to  file  a  protest  and  allows  for  recognition  of  the  factors  that 
can  be  associated  with  changes  in  the  protest  numbers. 

On  a  macro  level  (lawmakers,  SAF/AQCX),  the  study  provides  a  management 
tool  to  aid  in  forecasting  the  number  of  GAO  protests  for  a  particular  year.  An 
understanding  of  regression  analysis  or  complex  analysis  is  not  necessary;  decision¬ 
makers  should  recognize  the  legitimacy  of  a  variety  of  factors  affecting  the  number  of 
protests.  Instead  of  attributing  the  decline  of  protests  only  to  policy/law  changes,  the 
study  broadens  the  view  of  protests  with  previously  untested  variables.  A  decline  in  the 
number  of  protests  in  the  1 990s  may  be  due  to  fewer  contract  actions,  contract  dollars, 
and  relatively  positive  economic  times.  The  changes  brought  about  by  FASA  and  the 
FAR  15  rewrite  may  have  influenced  the  numbers;  however,  the  correlation  appears  to  be 
weak. 

The  future  conditions  will  provide  a  rigorous  test  for  the  changes  brought  about 
by  acquisition  reform.  If  the  number  of  AF  Contract  Actions  and  Dollars  increases  and  a 
general  economic  downturn  occurs,  will  the  number  of  protests  remain  at  the  lower 
levels?  How  long  will  the  downward  trend  continue?  The  data  (1984-1998)  indicate  that 
the  number  of  protests  may  rise  in  correspondence  to  these  other  factors;  however,  this 
assertion  goes  beyond  the  scope  of  this  study. 

On  a  micro  level,  the  study  does  not  suggest  contracting  officers  keep  a  daily 
watch  on  economic  indictors  to  determine  the  likelihood  of  a  protest.  On  the  other  hand, 
these  acquisition  professionals  should  be  aware  of  the  external  factors  (beyond  their 
control)  that  may  influence  the  protest  decision.  During  a  source  selection  or  bid 
opening,  the  contracting  officer  will  receive  the  inevitable  question:  what  are  the  chances 


of  receiving  a  protest?  The  answer  to  the  question  may  not  be  influenced  by  inequity  in 
the  bidding  process  or  improper  actions  by  the  agency.  This  study  acknowledges  the 
existence  of  the  other  factors  and  provides  acquisition  professionals  additional  resources 
to  view  the  likelihood  of  a  protest. 

The  contracting  officer’s  experience  and  knowledge  of  a  particular  situation  can 
combine  with  the  results  of  this  study  to  produce  a  powerful  tool.  During  the  course  of 
an  acquisition,  the  contracting  officer  may  be  able  to  conduct  a  risk  analysis  of  the 
possibility  of  receiving  the  protest  based  on  the  factors  outlined  in  the  study.  Instead  of 
taking  a  protest  avoidance  attitude,  the  agency  may  acknowledge  one  of  the  external 
factors  as  the  variable  that  pushes  contractor  into  the  protest  arena. 

Recommendations  For  Future  Study 

Based  on  the  results  of  this  study,  the  following  are  recommendations  for  future 
study: 

•  Employ  the  regression  model  using  different  independent  variables  such  as:  number 
of  AF  agency  protests,  number  of  defense  contractors,  etc. 

•  With  the  same  regression  model,  integrate  the  economic  impact  on  the  number  of 
protests  with  other  general  economic  conditions  (GNP,  inflation). 

•  Perform  a  statistical  analysis  of  AF  Protests  in  all  forums  (GAO,  Federal  Courts, 
Court  of  Claims)  to  increase  the  generalizability  of  this  model  and  broad  applicability 
of  the  results.  The  increased  number  of  forums/data  points  would  improve  the 
reliability  and  validity  of  statistical  analysis  in  this  area. 
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•  The  results  of  this  study  can  be  triangulated  with  qualitative  techniques  such  as 
telephone  surveys,  e-mail  surveys,  formal  surveys,  or  interviews  to  test  the  validity  of 
the  chosen  regression  model. 

•  With  a  qualitative  design,  send  out  a  survey  to  industry  (contractors)  to  determine  the 
other  factors  (economic  health  of  the  company,  high  level  support  for  the  RFP  team, 
quid  pro  qou)  that  motivate  the  decision  to  file  a  protest.  The  qualitative  portion  of 
the  research  will  serve  as  the  basis  for  a  statistical  analysis  of  the  factors. 

•  Perform  a  time-series  analysis  on  the  protest  data.  Such  a  study  would  require  at  least 
20  years  of  data  to  provide  a  meaningful  time-series  analysis.  The  researcher  must 
prove  that  the  years  are  dependent  upon  each  other  and  compare  various  periods  with 
each  other  for  variability. 

Limitations  of  the  Study 

The  data  gathered  from  SAF/AQCX  covers  AF  GAO-Protests  Processed  between 
1984-1998.  Any  predictions  beyond  this  period  in  another  protest  forum  bear  the  risk  of 
grossly  inaccurate  predictions  due  to  the  nature  of  the  regression  analysis  and  necessary 
assumptions. 

Conclusion 

Management  questions  regarding  acquisition  reform  and  external  factors  related 
to  the  number  of  protests  in  the  AF  were  addressed  in  this  study.  While  it  is  difficult  to 
quantify  the  impact  of  acquisition  reform  initiatives  on  the  amount  of  protests  filed,  the 
legitimacy  of  the  external  factors  was  established.  No  single  factor  could  be  labeled  as 
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the  cause  of  a  protest;  however,  certain  conditions  appear  to  contribute  to  the  trends  in  the 
number  of  protests  filed. 

While  care  has  been  taken  to  interpret  the  statistical  results,  equal  care  should  be 
taken  in  attributing  any  changes  to  policy  adjustment.  The  external  factors  appear  to 
affect  the  numbers  of  protests  filed  more  than  acquisition  reform  initiatives;  therefore,  AF 
acquisition  professionals  should  take  care  in  pointing  to  FAS  A  or  the  FAR  1 5  rewrite  as 
the  sole  cause  of  the  recent  decline. 
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